[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:29:48 dist-test-slave-fg3h krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:29:48 dist-test-slave-fg3h krb5kdc[399](info): set up 2 sockets
Jun 19 12:29:48 dist-test-slave-fg3h krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 12:29:49.389642 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:29:49 dist-test-slave-fg3h krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:29:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:29:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:29:49.516526 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36779
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:29:49.629057 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:29:49.629370 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:29:49.629442 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:29:49.633423 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:29:49.633515 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:29:49.633554 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:29:49.633617 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:29:49.633657 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:29:49.638827 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:29:49.639902 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:29:49.640970 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:29:49.649672 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:49.649689 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:49.649695 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:49.649904 412 server_base.cc:1034] running on GCE node
I20260619 12:29:49.650537 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:29:49.651647 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:29:49.652853 412 hybrid_clock.cc:648] HybridClock initialized: now 1781872189652802 us; error 65 us; skew 500 ppm
Jun 19 12:29:49 dist-test-slave-fg3h krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:29:49.659307 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:29:49.660262 412 webserver.cc:460] Webserver started at http://127.0.98.62:33903/ using document root <none> and password file <none>
I20260619 12:29:49.660861 412 fs_manager.cc:359] Metadata directory not provided
I20260619 12:29:49.660953 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:29:49.661242 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:29:49.663374 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "f4a45d3a62364333a231b5c141c7747e"
format_stamp: "Formatted at 2026-06-19 12:29:49 on dist-test-slave-fg3h"
server_key: "816bda5e9a10d1dc7f0d07b32ffc74aa"
server_key_iv: "2a83abe7e5b42a16808a6ebbcb397b9c"
server_key_version: "encryptionkey@0"
I20260619 12:29:49.663944 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "f4a45d3a62364333a231b5c141c7747e"
format_stamp: "Formatted at 2026-06-19 12:29:49 on dist-test-slave-fg3h"
server_key: "816bda5e9a10d1dc7f0d07b32ffc74aa"
server_key_iv: "2a83abe7e5b42a16808a6ebbcb397b9c"
server_key_version: "encryptionkey@0"
I20260619 12:29:49.667673 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 12:29:49.670068 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:49.671278 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:29:49.671429 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "f4a45d3a62364333a231b5c141c7747e"
format_stamp: "Formatted at 2026-06-19 12:29:49 on dist-test-slave-fg3h"
server_key: "816bda5e9a10d1dc7f0d07b32ffc74aa"
server_key_iv: "2a83abe7e5b42a16808a6ebbcb397b9c"
server_key_version: "encryptionkey@0"
I20260619 12:29:49.671543 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:29:49.685981 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:29:49.689074 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:29:49.689338 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:29:49.696278 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36779
I20260619 12:29:49.696296 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36779 every 8 connection(s)
I20260619 12:29:49.697304 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:29:49.700074 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:29:49.703243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 412
I20260619 12:29:49.703418 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:29:49.704695 392 external_mini_cluster.cc:1428] Setting key ab41f074b03afbf655272d9905d65e80
I20260619 12:29:49.705819 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e: Bootstrap starting.
I20260619 12:29:49.708565 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e: Neither blocks nor log segments found. Creating new log.
I20260619 12:29:49.709493 480 log.cc:824] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e: Log is configured to *not* fsync() on all Append() calls
I20260619 12:29:49.711295 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e: No bootstrap required, opened a new log
I20260619 12:29:49.715221 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a45d3a62364333a231b5c141c7747e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36779 } }
I20260619 12:29:49.715458 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:29:49.715520 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4a45d3a62364333a231b5c141c7747e, State: Initialized, Role: FOLLOWER
I20260619 12:29:49.715978 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a45d3a62364333a231b5c141c7747e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36779 } }
I20260619 12:29:49.716102 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:29:49.716183 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:29:49.716315 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:29:49.717288 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a45d3a62364333a231b5c141c7747e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36779 } }
I20260619 12:29:49.717626 480 leader_election.cc:302] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4a45d3a62364333a231b5c141c7747e; no voters:
I20260619 12:29:49.717965 480 leader_election.cc:288] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:29:49.718073 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:29:49.718248 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e [term 1 LEADER]: Becoming Leader. State: Replica: f4a45d3a62364333a231b5c141c7747e, State: Running, Role: LEADER
I20260619 12:29:49.718596 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a45d3a62364333a231b5c141c7747e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36779 } }
I20260619 12:29:49.719249 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:29:49.719771 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4a45d3a62364333a231b5c141c7747e. Latest consensus state: current_term: 1 leader_uuid: "f4a45d3a62364333a231b5c141c7747e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a45d3a62364333a231b5c141c7747e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36779 } } }
I20260619 12:29:49.719904 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e [sys.catalog]: This master's current role is: LEADER
I20260619 12:29:49.720605 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4a45d3a62364333a231b5c141c7747e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a45d3a62364333a231b5c141c7747e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36779 } } }
I20260619 12:29:49.720711 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e [sys.catalog]: This master's current role is: LEADER
I20260619 12:29:49.721060 490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 12:29:49 dist-test-slave-fg3h krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:29:49.724390 490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:29:49.730036 490 catalog_manager.cc:1347] Generated new cluster ID: 305a1603d90f45d9bb5793a08ee426c4
I20260619 12:29:49.730098 490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:29:49.731792 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:49.707906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.0.98.62:36779)
0619 12:29:49.708327 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:49.708342 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:49.709500 (+ 1158us) server_negotiation.cc:407] Connection header received
0619 12:29:49.713460 (+ 3960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:49.713478 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:49.713798 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:49.714112 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:49.716485 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:49.717588 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:49.719136 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:49.719860 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:49.720019 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:49.725253 (+ 5234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:49.725261 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:49.725276 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:49.725314 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:49.727488 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:49.728147 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:49.728152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:49.728157 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:49.728307 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:49.729023 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:49.729027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:49.729029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:49.729559 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:49.729790 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:49.730672 (+ 882us) server_negotiation.cc:299] Negotiation successful
0619 12:29:49.730965 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":148,"threads_started":1}
I20260619 12:29:49.740947 490 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:29:49.742126 490 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:29:49.748039 490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4a45d3a62364333a231b5c141c7747e: Generated new TSK 0
I20260619 12:29:49.748719 490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:29:49.778517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36779
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:29:49.908828 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:29:49.909139 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:29:49.909220 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:29:49.913277 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:29:49.913337 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:29:49.913398 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:29:49.918808 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36779
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:29:49.919919 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:29:49.921026 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:29:49.930630 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:49.930800 506 server_base.cc:1034] running on GCE node
W20260619 12:29:49.930714 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:49.930630 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:49.931294 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:29:49.931946 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:29:49.933175 506 hybrid_clock.cc:648] HybridClock initialized: now 1781872189933161 us; error 38 us; skew 500 ppm
Jun 19 12:29:49 dist-test-slave-fg3h krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:29:49.936106 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:29:49.936990 506 webserver.cc:460] Webserver started at http://127.0.98.1:37563/ using document root <none> and password file <none>
I20260619 12:29:49.937595 506 fs_manager.cc:359] Metadata directory not provided
I20260619 12:29:49.937693 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:29:49.937930 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:29:49.939998 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "2c38d7f0d6ab4b0a834abac1b2abc562"
format_stamp: "Formatted at 2026-06-19 12:29:49 on dist-test-slave-fg3h"
server_key: "cc766935839965e3c7c26bab2e8411b7"
server_key_iv: "d6535671a703b971f9218a08539a594d"
server_key_version: "encryptionkey@0"
I20260619 12:29:49.940610 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2c38d7f0d6ab4b0a834abac1b2abc562"
format_stamp: "Formatted at 2026-06-19 12:29:49 on dist-test-slave-fg3h"
server_key: "cc766935839965e3c7c26bab2e8411b7"
server_key_iv: "d6535671a703b971f9218a08539a594d"
server_key_version: "encryptionkey@0"
I20260619 12:29:49.944494 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:29:49.947048 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:49.948339 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:29:49.948485 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "2c38d7f0d6ab4b0a834abac1b2abc562"
format_stamp: "Formatted at 2026-06-19 12:29:49 on dist-test-slave-fg3h"
server_key: "cc766935839965e3c7c26bab2e8411b7"
server_key_iv: "d6535671a703b971f9218a08539a594d"
server_key_version: "encryptionkey@0"
I20260619 12:29:49.948613 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:29:49.987532 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:29:49.991127 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:29:49.991464 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:29:49.992245 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:29:49.993271 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:29:49.993355 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:49.993427 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:29:49.993477 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:50.005079 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40941
I20260619 12:29:50.005111 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40941 every 8 connection(s)
I20260619 12:29:50.006280 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:29:50.016338 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 506
I20260619 12:29:50.016477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:29:50.016811 392 external_mini_cluster.cc:1428] Setting key e65c431fa9b34fc9ede8418104ae3b9d
Jun 19 12:29:50 dist-test-slave-fg3h krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:29:50.023537 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:50.008928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58241 (local address 127.0.98.62:36779)
0619 12:29:50.009261 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:50.009267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:50.010397 (+ 1130us) server_negotiation.cc:407] Connection header received
0619 12:29:50.011884 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:50.011888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:50.011949 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:50.012040 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:50.013429 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:50.014058 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:50.015259 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:50.015495 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:50.015607 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:50.018871 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:50.018877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:50.018885 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:50.018909 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:50.020832 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:50.021635 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:50.021638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:50.021639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:50.021727 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:50.022150 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:50.022157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:50.022162 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:50.022449 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:50.022565 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:50.023129 (+ 564us) server_negotiation.cc:299] Negotiation successful
0619 12:29:50.023290 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260619 12:29:50.024459 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:50.009639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36779 (local address 127.0.98.1:58241)
0619 12:29:50.010208 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:50.010258 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:50.011631 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:50.012266 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:50.012274 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:50.012667 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:29:50.013275 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:50.013291 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:50.014208 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:50.014211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:50.015117 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:50.015127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:50.015665 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:50.015669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:50.015846 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:50.016612 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:29:50.016635 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:29:50.018584 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:29:50.020967 (+ 2383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:50.020978 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:50.020999 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:50.021477 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:50.021829 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:50.021832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:50.021833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:50.022017 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:50.022566 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:29:50.022572 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:29:50.022846 (+ 274us) client_negotiation.cc:769] Sending connection context
0619 12:29:50.023012 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 12:29:50.023374 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":176,"threads_started":1}
I20260619 12:29:50.025943 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36779
I20260619 12:29:50.026237 635 heartbeater.cc:464] Registering TS with master...
I20260619 12:29:50.026895 635 heartbeater.cc:511] Master 127.0.98.62:36779 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:29:50.028570 444 ts_manager.cc:194] Registered new tserver with Master: 2c38d7f0d6ab4b0a834abac1b2abc562 (127.0.98.1:40941)
I20260619 12:29:50.029922 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:29:50.049964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36779
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:29:50.161801 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:29:50.162083 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:29:50.162129 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:29:50.166385 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:29:50.166450 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:29:50.166543 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:29:50.171880 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36779
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:29:50.172945 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:29:50.174003 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:29:50.183722 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:50.183729 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:50.183825 640 server_base.cc:1034] running on GCE node
W20260619 12:29:50.183729 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:50.184384 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:29:50.184933 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:29:50.186105 640 hybrid_clock.cc:648] HybridClock initialized: now 1781872190186074 us; error 46 us; skew 500 ppm
Jun 19 12:29:50 dist-test-slave-fg3h krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:29:50.188679 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:29:50.189528 640 webserver.cc:460] Webserver started at http://127.0.98.2:36119/ using document root <none> and password file <none>
I20260619 12:29:50.190037 640 fs_manager.cc:359] Metadata directory not provided
I20260619 12:29:50.190093 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:29:50.190259 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:29:50.192212 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "7dec114465904c6bba6ac3b1ff4eb754"
format_stamp: "Formatted at 2026-06-19 12:29:50 on dist-test-slave-fg3h"
server_key: "88da3956d0aa72a76ce13f9be64b046c"
server_key_iv: "2659d74710ae4791e6c633cbc6d0f928"
server_key_version: "encryptionkey@0"
I20260619 12:29:50.192718 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7dec114465904c6bba6ac3b1ff4eb754"
format_stamp: "Formatted at 2026-06-19 12:29:50 on dist-test-slave-fg3h"
server_key: "88da3956d0aa72a76ce13f9be64b046c"
server_key_iv: "2659d74710ae4791e6c633cbc6d0f928"
server_key_version: "encryptionkey@0"
I20260619 12:29:50.196238 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 12:29:50.198673 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:50.199890 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:29:50.200021 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "7dec114465904c6bba6ac3b1ff4eb754"
format_stamp: "Formatted at 2026-06-19 12:29:50 on dist-test-slave-fg3h"
server_key: "88da3956d0aa72a76ce13f9be64b046c"
server_key_iv: "2659d74710ae4791e6c633cbc6d0f928"
server_key_version: "encryptionkey@0"
I20260619 12:29:50.200129 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:29:50.211553 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:29:50.214761 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:29:50.214956 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:29:50.215512 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:29:50.216305 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:29:50.216375 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:50.216446 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:29:50.216486 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:50.226552 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38485
I20260619 12:29:50.226581 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38485 every 8 connection(s)
I20260619 12:29:50.227442 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 12:29:50 dist-test-slave-fg3h krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:29:50.236788 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 640
I20260619 12:29:50.236917 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:29:50.237246 392 external_mini_cluster.cc:1428] Setting key a2f0137cfa80588d46cb15b1cc612e46
I20260619 12:29:50.240926 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:50.229210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33567 (local address 127.0.98.62:36779)
0619 12:29:50.229371 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:50.229376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:50.230293 (+ 917us) server_negotiation.cc:407] Connection header received
0619 12:29:50.231595 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:50.231598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:50.231644 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:50.231720 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:50.232907 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:50.233494 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:50.234620 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:50.234842 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:50.234926 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:50.237416 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:50.237424 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:50.237427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:50.237451 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:50.238903 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:50.239422 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:50.239427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:50.239429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:50.239509 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:50.239852 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:50.239856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:50.239858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:50.240041 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:50.240147 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:50.240631 (+ 484us) server_negotiation.cc:299] Negotiation successful
0619 12:29:50.240771 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:29:50.241827 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:50.229684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36779 (local address 127.0.98.2:33567)
0619 12:29:50.230141 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:50.230184 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:50.231399 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:50.231936 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:50.231943 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:50.232297 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:29:50.232775 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:50.232788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:50.233603 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:50.233605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:50.234513 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:50.234522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:50.234967 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:50.234970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:50.235100 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:50.235678 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:29:50.235695 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:29:50.237241 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:29:50.239037 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:50.239043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:50.239057 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:50.239321 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:50.239608 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:50.239611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:50.239613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:50.239749 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:50.240144 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:29:50.240152 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:29:50.240394 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 12:29:50.240595 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 12:29:50.240921 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
I20260619 12:29:50.243093 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36779
I20260619 12:29:50.243366 769 heartbeater.cc:464] Registering TS with master...
I20260619 12:29:50.243968 769 heartbeater.cc:511] Master 127.0.98.62:36779 requested a full tablet report, sending...
I20260619 12:29:50.245160 444 ts_manager.cc:194] Registered new tserver with Master: 7dec114465904c6bba6ac3b1ff4eb754 (127.0.98.2:38485)
I20260619 12:29:50.245983 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33567
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:29:50.264685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36779
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:29:50.377005 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:29:50.377344 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:29:50.377391 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:29:50.381645 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:29:50.381702 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:29:50.381789 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:29:50.387151 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36779
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:29:50.388247 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:29:50.389339 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:29:50.398831 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:50.398945 774 server_base.cc:1034] running on GCE node
W20260619 12:29:50.398825 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:50.398900 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:50.399618 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:29:50.400282 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:29:50.401441 774 hybrid_clock.cc:648] HybridClock initialized: now 1781872190401404 us; error 50 us; skew 500 ppm
Jun 19 12:29:50 dist-test-slave-fg3h krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:29:50.404336 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:29:50.405354 774 webserver.cc:460] Webserver started at http://127.0.98.3:35685/ using document root <none> and password file <none>
I20260619 12:29:50.405905 774 fs_manager.cc:359] Metadata directory not provided
I20260619 12:29:50.405962 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:29:50.406126 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:29:50.408097 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "ab29705ce3a74756904eb7ff1356a34e"
format_stamp: "Formatted at 2026-06-19 12:29:50 on dist-test-slave-fg3h"
server_key: "6b4456ecb074ba8836c3d8c2bce89a1c"
server_key_iv: "c06ee8a38d386f86388dfe2503707b60"
server_key_version: "encryptionkey@0"
I20260619 12:29:50.408623 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ab29705ce3a74756904eb7ff1356a34e"
format_stamp: "Formatted at 2026-06-19 12:29:50 on dist-test-slave-fg3h"
server_key: "6b4456ecb074ba8836c3d8c2bce89a1c"
server_key_iv: "c06ee8a38d386f86388dfe2503707b60"
server_key_version: "encryptionkey@0"
I20260619 12:29:50.412410 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:29:50.415037 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:50.416416 774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 12:29:50.416574 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "ab29705ce3a74756904eb7ff1356a34e"
format_stamp: "Formatted at 2026-06-19 12:29:50 on dist-test-slave-fg3h"
server_key: "6b4456ecb074ba8836c3d8c2bce89a1c"
server_key_iv: "c06ee8a38d386f86388dfe2503707b60"
server_key_version: "encryptionkey@0"
I20260619 12:29:50.416702 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:29:50.434078 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:29:50.437052 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:29:50.437323 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:29:50.437919 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:29:50.438726 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:29:50.438797 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:50.438866 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:29:50.438915 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:50.449064 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45743
I20260619 12:29:50.449141 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45743 every 8 connection(s)
I20260619 12:29:50.450063 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:29:50.451561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 774
I20260619 12:29:50.451684 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:29:50.451985 392 external_mini_cluster.cc:1428] Setting key 416e7cc69a5e90a21ce9f2e896c2b036
Jun 19 12:29:50 dist-test-slave-fg3h krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:29:50.464851 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:50.452065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36057 (local address 127.0.98.62:36779)
0619 12:29:50.452186 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:50.452191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:50.453282 (+ 1091us) server_negotiation.cc:407] Connection header received
0619 12:29:50.455043 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:50.455046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:50.455098 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:50.455174 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:50.456370 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:50.456905 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:50.458097 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:50.458339 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:50.458458 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:50.460933 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:50.460938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:50.460945 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:50.460970 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:50.462483 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:50.463137 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:50.463141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:50.463143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:50.463235 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:50.463634 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:50.463640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:50.463642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:50.463852 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:50.463970 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:50.464510 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 12:29:50.464656 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 12:29:50.465958 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:50.452544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36779 (local address 127.0.98.3:36057)
0619 12:29:50.453147 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:50.453203 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:50.454838 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:50.455345 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:50.455351 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:50.455700 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:29:50.456236 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:50.456250 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:50.457028 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:50.457031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:50.457969 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:50.457979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:50.458497 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:50.458501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:50.458672 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:50.459280 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:29:50.459295 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:29:50.460763 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:29:50.462666 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:50.462673 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:50.462690 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:50.463007 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:50.463379 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:50.463383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:50.463384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:50.463526 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:50.463982 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:29:50.463991 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:29:50.464248 (+ 257us) client_negotiation.cc:769] Sending connection context
0619 12:29:50.464465 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 12:29:50.464813 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":164,"threads_started":1}
I20260619 12:29:50.467197 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36779
I20260619 12:29:50.467478 903 heartbeater.cc:464] Registering TS with master...
I20260619 12:29:50.468025 903 heartbeater.cc:511] Master 127.0.98.62:36779 requested a full tablet report, sending...
I20260619 12:29:50.469193 443 ts_manager.cc:194] Registered new tserver with Master: ab29705ce3a74756904eb7ff1356a34e (127.0.98.3:45743)
I20260619 12:29:50.469784 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36057
I20260619 12:29:50.478785 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 12:29:50 dist-test-slave-fg3h krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872190, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 12:29:50 dist-test-slave-fg3h krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872190, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:29:50.506582 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:50.496864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43598 (local address 127.0.98.1:40941)
0619 12:29:50.497209 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:50.497228 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:50.497300 (+ 72us) server_negotiation.cc:407] Connection header received
0619 12:29:50.497474 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:50.497480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:50.497671 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:50.497827 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:50.498488 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:50.499237 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:50.500128 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:50.500505 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:50.500628 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:50.503176 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:50.503184 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:50.503197 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:50.503219 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:50.504811 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:50.505261 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:50.505265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:50.505266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:50.505356 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:50.505693 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:50.505695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:50.505697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:50.505943 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:50.506094 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:50.506300 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 12:29:50.506430 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":116,"threads_started":1}
W20260619 12:29:50.507682 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43598
Jun 19 12:29:50 dist-test-slave-fg3h krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 12:29:50 dist-test-slave-fg3h krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:29:50.533694 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:50.525432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.0.98.1:40941)
0619 12:29:50.525635 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:50.525640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:50.525774 (+ 134us) server_negotiation.cc:407] Connection header received
0619 12:29:50.525948 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:50.525952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:50.526004 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:50.526100 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:50.526798 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:50.527319 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:50.528160 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:50.528398 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:50.528518 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:50.530950 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:50.530953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:50.530956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:50.530982 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:50.532219 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:50.532599 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:50.532602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:50.532603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:50.532672 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:50.532888 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:50.532890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:50.532891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:50.533041 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:50.533208 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:50.533474 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 12:29:50.533572 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 12:29:50.534813 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 506
I20260619 12:29:50.540593 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 640
I20260619 12:29:50.548089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 774
I20260619 12:29:50.553354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 412
2026-06-19T12:29:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2227 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:29:50 dist-test-slave-fg3h krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:29:50 dist-test-slave-fg3h krb5kdc[917](info): set up 2 sockets
Jun 19 12:29:50 dist-test-slave-fg3h krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260619 12:29:51.640681 392 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:29:51 dist-test-slave-fg3h krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:29:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:29:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:29:51.764812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32999
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:29:51.877609 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:29:51.877887 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:29:51.877959 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:29:51.882144 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:29:51.882207 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:29:51.882227 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:29:51.882244 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:29:51.882259 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:29:51.887797 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:29:51.888890 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:29:51.890038 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:29:51.898662 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:51.898701 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:51.898662 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:51.899071 930 server_base.cc:1034] running on GCE node
I20260619 12:29:51.899672 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:29:51.900818 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:29:51.902006 930 hybrid_clock.cc:648] HybridClock initialized: now 1781872191901970 us; error 51 us; skew 500 ppm
Jun 19 12:29:51 dist-test-slave-fg3h krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:29:51.904948 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:29:51.905963 930 webserver.cc:460] Webserver started at http://127.0.98.62:36909/ using document root <none> and password file <none>
I20260619 12:29:51.906539 930 fs_manager.cc:359] Metadata directory not provided
I20260619 12:29:51.906630 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:29:51.906869 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:29:51.908941 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "f060950ed8474348a2c823b093d76eef"
format_stamp: "Formatted at 2026-06-19 12:29:51 on dist-test-slave-fg3h"
server_key: "d0b51d67ecf5931f90b542e65b544131"
server_key_iv: "dccf072e165a4e7299821ed84209e855"
server_key_version: "encryptionkey@0"
I20260619 12:29:51.909582 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "f060950ed8474348a2c823b093d76eef"
format_stamp: "Formatted at 2026-06-19 12:29:51 on dist-test-slave-fg3h"
server_key: "d0b51d67ecf5931f90b542e65b544131"
server_key_iv: "dccf072e165a4e7299821ed84209e855"
server_key_version: "encryptionkey@0"
I20260619 12:29:51.913439 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:29:51.915946 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:51.917230 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:29:51.917393 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "f060950ed8474348a2c823b093d76eef"
format_stamp: "Formatted at 2026-06-19 12:29:51 on dist-test-slave-fg3h"
server_key: "d0b51d67ecf5931f90b542e65b544131"
server_key_iv: "dccf072e165a4e7299821ed84209e855"
server_key_version: "encryptionkey@0"
I20260619 12:29:51.917507 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:29:51.940575 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:29:51.944067 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:29:51.944278 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:29:51.951318 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32999
I20260619 12:29:51.951364 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32999 every 8 connection(s)
I20260619 12:29:51.952265 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:29:51.954981 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:29:51.960377 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef: Bootstrap starting.
I20260619 12:29:51.961469 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 930
I20260619 12:29:51.961588 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:29:51.961866 392 external_mini_cluster.cc:1428] Setting key fa9f374dc6dfb935ba9f68cc717e6b1b
I20260619 12:29:51.963380 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef: Neither blocks nor log segments found. Creating new log.
I20260619 12:29:51.964107 998 log.cc:824] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef: Log is configured to *not* fsync() on all Append() calls
I20260619 12:29:51.966130 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef: No bootstrap required, opened a new log
I20260619 12:29:51.970546 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060950ed8474348a2c823b093d76eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32999 } }
I20260619 12:29:51.970826 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:29:51.970888 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f060950ed8474348a2c823b093d76eef, State: Initialized, Role: FOLLOWER
I20260619 12:29:51.971441 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060950ed8474348a2c823b093d76eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32999 } }
I20260619 12:29:51.971618 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:29:51.971683 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:29:51.971776 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef [term 0 FOLLOWER]: Advancing to term 1
Jun 19 12:29:51 dist-test-slave-fg3h krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:29:51.972942 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060950ed8474348a2c823b093d76eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32999 } }
I20260619 12:29:51.973330 998 leader_election.cc:302] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f060950ed8474348a2c823b093d76eef; no voters:
I20260619 12:29:51.973677 998 leader_election.cc:288] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:29:51.973762 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:29:51.973989 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef [term 1 LEADER]: Becoming Leader. State: Replica: f060950ed8474348a2c823b093d76eef, State: Running, Role: LEADER
I20260619 12:29:51.974289 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060950ed8474348a2c823b093d76eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32999 } }
I20260619 12:29:51.975010 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:29:51.975469 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef [sys.catalog]: SysCatalogTable state changed. Reason: New leader f060950ed8474348a2c823b093d76eef. Latest consensus state: current_term: 1 leader_uuid: "f060950ed8474348a2c823b093d76eef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060950ed8474348a2c823b093d76eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32999 } } }
I20260619 12:29:51.975585 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef [sys.catalog]: This master's current role is: LEADER
I20260619 12:29:51.975881 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f060950ed8474348a2c823b093d76eef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f060950ed8474348a2c823b093d76eef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32999 } } }
I20260619 12:29:51.975988 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef [sys.catalog]: This master's current role is: LEADER
I20260619 12:29:51.975991 1007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:29:51.979035 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:51.963102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58354 (local address 127.0.98.62:32999)
0619 12:29:51.963613 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:51.963631 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:51.963669 (+ 38us) server_negotiation.cc:407] Connection header received
0619 12:29:51.964449 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:51.964477 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:51.964926 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:51.965375 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:51.966231 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:51.967364 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:51.968300 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:51.968846 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:51.969066 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:51.972602 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:51.972609 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:51.972626 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:51.972667 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:51.974842 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:51.975402 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:51.975408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:51.975414 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:51.975553 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:51.976219 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:51.976222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:51.976223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:51.976697 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:51.977631 (+ 934us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:51.977794 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 12:29:51.978093 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":141,"threads_started":1}
I20260619 12:29:51.979444 1007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:29:51.985252 1007 catalog_manager.cc:1347] Generated new cluster ID: 10d90cea6456471888f8cec5fd767eaf
I20260619 12:29:51.985332 1007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:29:52.004318 1007 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:29:52.005527 1007 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:29:52.014850 1007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f060950ed8474348a2c823b093d76eef: Generated new TSK 0
I20260619 12:29:52.015626 1007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:29:52.047906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32999
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:29:52.159683 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:29:52.159963 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:29:52.160008 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:29:52.164201 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:29:52.164271 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:29:52.164335 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:29:52.169876 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32999
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:29:52.170933 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:29:52.171911 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:29:52.181639 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:52.181633 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:52.181633 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:52.181727 1024 server_base.cc:1034] running on GCE node
I20260619 12:29:52.182323 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:29:52.182978 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:29:52.184195 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781872192184138 us; error 77 us; skew 500 ppm
Jun 19 12:29:52 dist-test-slave-fg3h krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:29:52.187078 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:29:52.187963 1024 webserver.cc:460] Webserver started at http://127.0.98.1:41659/ using document root <none> and password file <none>
I20260619 12:29:52.188566 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 12:29:52.188660 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:29:52.188889 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:29:52.190982 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "f3ba6e740ba4450a9ce1305d0d5618e0"
format_stamp: "Formatted at 2026-06-19 12:29:52 on dist-test-slave-fg3h"
server_key: "980b40cd143da029503799ae0f6dc395"
server_key_iv: "907a1155a91b7a12c1d0617894c3c2c8"
server_key_version: "encryptionkey@0"
I20260619 12:29:52.191568 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f3ba6e740ba4450a9ce1305d0d5618e0"
format_stamp: "Formatted at 2026-06-19 12:29:52 on dist-test-slave-fg3h"
server_key: "980b40cd143da029503799ae0f6dc395"
server_key_iv: "907a1155a91b7a12c1d0617894c3c2c8"
server_key_version: "encryptionkey@0"
I20260619 12:29:52.195411 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:29:52.198045 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:52.199436 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:29:52.199590 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "f3ba6e740ba4450a9ce1305d0d5618e0"
format_stamp: "Formatted at 2026-06-19 12:29:52 on dist-test-slave-fg3h"
server_key: "980b40cd143da029503799ae0f6dc395"
server_key_iv: "907a1155a91b7a12c1d0617894c3c2c8"
server_key_version: "encryptionkey@0"
I20260619 12:29:52.199712 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:29:52.210923 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:29:52.214257 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:29:52.214483 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:29:52.215086 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:29:52.215940 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:29:52.216012 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:52.216078 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:29:52.216126 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:52.225769 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42129
I20260619 12:29:52.225791 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42129 every 8 connection(s)
I20260619 12:29:52.226645 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:29:52.234230 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 1024
I20260619 12:29:52.234346 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:29:52.234614 392 external_mini_cluster.cc:1428] Setting key b2216ae73e178a037a1db3842547e9bf
Jun 19 12:29:52 dist-test-slave-fg3h krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:29:52.241245 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:52.228581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41175 (local address 127.0.98.62:32999)
0619 12:29:52.228715 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:52.228719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:52.229609 (+ 890us) server_negotiation.cc:407] Connection header received
0619 12:29:52.230931 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:52.230934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:52.230980 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:52.231054 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:52.232209 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:52.232726 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:52.233907 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:52.234130 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:52.234212 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:52.237031 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:52.237034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:52.237036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:52.237058 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:52.239073 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:52.239608 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:52.239613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:52.239615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:52.239702 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:52.240063 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:52.240066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:52.240068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:52.240261 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:52.240373 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:52.240874 (+ 501us) server_negotiation.cc:299] Negotiation successful
0619 12:29:52.241014 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:29:52.242031 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:52.228958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32999 (local address 127.0.98.1:41175)
0619 12:29:52.229460 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:52.229502 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:52.230726 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:52.231243 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:52.231250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:52.231601 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:29:52.232068 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:52.232081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:52.232854 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:52.232856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:52.233797 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:52.233807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:52.234337 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:52.234340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:52.234541 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:52.235148 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:29:52.235164 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:29:52.236847 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:29:52.239192 (+ 2345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:52.239199 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:52.239215 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:52.239498 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:52.239804 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:52.239809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:52.239811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:52.239959 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:52.240369 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:29:52.240377 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:29:52.240639 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 12:29:52.240834 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 12:29:52.241206 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":172,"threads_started":1}
I20260619 12:29:52.243393 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32999
I20260619 12:29:52.243638 1153 heartbeater.cc:464] Registering TS with master...
I20260619 12:29:52.244136 1153 heartbeater.cc:511] Master 127.0.98.62:32999 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:29:52.245749 962 ts_manager.cc:194] Registered new tserver with Master: f3ba6e740ba4450a9ce1305d0d5618e0 (127.0.98.1:42129)
I20260619 12:29:52.246876 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:29:52.264799 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32999
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:29:52.380400 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:29:52.380676 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:29:52.380757 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:29:52.385504 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:29:52.385602 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:29:52.385703 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:29:52.391876 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32999
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:29:52.393004 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:29:52.394148 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:29:52.404568 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:52.404594 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:52.404570 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:52.404767 1158 server_base.cc:1034] running on GCE node
I20260619 12:29:52.405587 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:29:52.406226 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:29:52.407405 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781872192407380 us; error 39 us; skew 500 ppm
Jun 19 12:29:52 dist-test-slave-fg3h krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:29:52.411149 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:29:52.412156 1158 webserver.cc:460] Webserver started at http://127.0.98.2:41899/ using document root <none> and password file <none>
I20260619 12:29:52.412744 1158 fs_manager.cc:359] Metadata directory not provided
I20260619 12:29:52.412838 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:29:52.413076 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:29:52.415267 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "0e7fd94581434e3a83abf1216ee8c737"
format_stamp: "Formatted at 2026-06-19 12:29:52 on dist-test-slave-fg3h"
server_key: "d457464d53af8feb8d092b52a9bbfea3"
server_key_iv: "0b12af82178fbccf31e42786dd238994"
server_key_version: "encryptionkey@0"
I20260619 12:29:52.415824 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0e7fd94581434e3a83abf1216ee8c737"
format_stamp: "Formatted at 2026-06-19 12:29:52 on dist-test-slave-fg3h"
server_key: "d457464d53af8feb8d092b52a9bbfea3"
server_key_iv: "0b12af82178fbccf31e42786dd238994"
server_key_version: "encryptionkey@0"
I20260619 12:29:52.419708 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:29:52.422595 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:52.424193 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 12:29:52.424340 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "0e7fd94581434e3a83abf1216ee8c737"
format_stamp: "Formatted at 2026-06-19 12:29:52 on dist-test-slave-fg3h"
server_key: "d457464d53af8feb8d092b52a9bbfea3"
server_key_iv: "0b12af82178fbccf31e42786dd238994"
server_key_version: "encryptionkey@0"
I20260619 12:29:52.424492 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:29:52.458899 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:29:52.462697 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:29:52.462901 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:29:52.463549 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:29:52.464409 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:29:52.464458 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:52.464526 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:29:52.464560 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:52.473982 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40887
I20260619 12:29:52.474010 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40887 every 8 connection(s)
I20260619 12:29:52.474958 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:29:52.481976 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 1158
I20260619 12:29:52.482087 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:29:52.482357 392 external_mini_cluster.cc:1428] Setting key fe7d6c677985a5c1a72301788391d489
Jun 19 12:29:52 dist-test-slave-fg3h krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:29:52.491638 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:52.477242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49255 (local address 127.0.98.62:32999)
0619 12:29:52.477417 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:52.477422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:52.478511 (+ 1089us) server_negotiation.cc:407] Connection header received
0619 12:29:52.479833 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:52.479839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:52.479897 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:52.479992 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:52.481261 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:52.481892 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:52.483332 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:52.483655 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:52.483770 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:52.487303 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:52.487306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:52.487308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:52.487329 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:52.488923 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:52.489670 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:52.489673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:52.489674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:52.489786 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:52.490241 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:52.490245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:52.490246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:52.490440 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:52.490543 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:52.491229 (+ 686us) server_negotiation.cc:299] Negotiation successful
0619 12:29:52.491343 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:29:52.492749 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:52.477791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32999 (local address 127.0.98.2:49255)
0619 12:29:52.478351 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:52.478395 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:52.479601 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:52.480152 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:52.480159 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:52.480563 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:29:52.481036 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:52.481049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:52.482034 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:52.482038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:52.483209 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:52.483217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:52.483875 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:52.483879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:52.484085 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:52.484737 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:29:52.484762 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:29:52.487093 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:29:52.489052 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:52.489062 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:52.489108 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:52.489524 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:52.489921 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:52.489927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:52.489930 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:52.490131 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:52.490550 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:29:52.490557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:29:52.490922 (+ 365us) client_negotiation.cc:769] Sending connection context
0619 12:29:52.491163 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 12:29:52.491535 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":126,"threads_started":1}
I20260619 12:29:52.493916 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32999
I20260619 12:29:52.494166 1287 heartbeater.cc:464] Registering TS with master...
I20260619 12:29:52.494690 1287 heartbeater.cc:511] Master 127.0.98.62:32999 requested a full tablet report, sending...
I20260619 12:29:52.495862 962 ts_manager.cc:194] Registered new tserver with Master: 0e7fd94581434e3a83abf1216ee8c737 (127.0.98.2:40887)
I20260619 12:29:52.496408 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 12:29:53.249678 1153 heartbeater.cc:503] Master 127.0.98.62:32999 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260619 12:29:53.326862 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.843s user 0.000s sys 0.003s
I20260619 12:29:53.327371 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32999
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:29:53.454605 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:29:53.454926 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:29:53.455011 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:29:53.459882 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:29:53.459955 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:29:53.460059 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:29:53.466671 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32999
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:29:53.467934 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:29:53.469137 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:29:53.480394 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:53.480408 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:53.480468 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:53.480767 1292 server_base.cc:1034] running on GCE node
I20260619 12:29:53.481417 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:29:53.482231 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:29:53.483402 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781872193483380 us; error 35 us; skew 500 ppm
Jun 19 12:29:53 dist-test-slave-fg3h krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:29:53.486555 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:29:53.487510 1292 webserver.cc:460] Webserver started at http://127.0.98.3:38817/ using document root <none> and password file <none>
I20260619 12:29:53.488140 1292 fs_manager.cc:359] Metadata directory not provided
I20260619 12:29:53.488209 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:29:53.488381 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:29:53.490743 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "9c348efddd264229bec23e2eab0fae6a"
format_stamp: "Formatted at 2026-06-19 12:29:53 on dist-test-slave-fg3h"
server_key: "b1e8b3ae7799abbe6b76502a20e1a86d"
server_key_iv: "4ec53cb5be686a7e57732b91e84d08cf"
server_key_version: "encryptionkey@0"
I20260619 12:29:53.491276 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9c348efddd264229bec23e2eab0fae6a"
format_stamp: "Formatted at 2026-06-19 12:29:53 on dist-test-slave-fg3h"
server_key: "b1e8b3ae7799abbe6b76502a20e1a86d"
server_key_iv: "4ec53cb5be686a7e57732b91e84d08cf"
server_key_version: "encryptionkey@0"
I20260619 12:29:53.495762 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:29:53.498421 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:53.498430 1287 heartbeater.cc:503] Master 127.0.98.62:32999 was elected leader, sending a full tablet report...
I20260619 12:29:53.500381 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 12:29:53.500535 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "9c348efddd264229bec23e2eab0fae6a"
format_stamp: "Formatted at 2026-06-19 12:29:53 on dist-test-slave-fg3h"
server_key: "b1e8b3ae7799abbe6b76502a20e1a86d"
server_key_iv: "4ec53cb5be686a7e57732b91e84d08cf"
server_key_version: "encryptionkey@0"
I20260619 12:29:53.500666 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:29:53.516053 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:29:53.519898 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:29:53.520107 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:29:53.520717 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:29:53.521733 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:29:53.521782 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:53.521848 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:29:53.521888 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:53.531702 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44877
I20260619 12:29:53.531777 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44877 every 8 connection(s)
I20260619 12:29:53.532742 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:29:53.534693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 1292
I20260619 12:29:53.534826 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:29:53.535087 392 external_mini_cluster.cc:1428] Setting key 9bc299845db38194415c7a000acb8247
Jun 19 12:29:53 dist-test-slave-fg3h krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:29:53.547417 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:53.534722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43021 (local address 127.0.98.62:32999)
0619 12:29:53.535029 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:53.535034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:53.536056 (+ 1022us) server_negotiation.cc:407] Connection header received
0619 12:29:53.537674 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:53.537677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:53.537729 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:53.537813 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:53.538970 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.539517 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.540670 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.540908 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.541022 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:53.543761 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:53.543764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:53.543767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:53.543790 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:53.545355 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:53.545859 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:53.545864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:53.545866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:53.545952 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:53.546306 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:53.546310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:53.546312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:53.546496 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:53.546604 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:53.547110 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 12:29:53.547236 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":101,"threads_started":1}
I20260619 12:29:53.548343 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:53.535240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32999 (local address 127.0.98.3:43021)
0619 12:29:53.535809 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:53.535860 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:53.537474 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:53.537980 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:53.537988 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:53.538352 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:29:53.538821 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:53.538835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.539665 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.539674 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:53.540547 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:53.540555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.541055 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.541059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:53.541257 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:53.541867 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:29:53.541883 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:29:53.543559 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:29:53.545471 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:53.545478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:53.545492 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:53.545755 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:53.546047 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:53.546051 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:53.546053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:53.546210 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:53.546612 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:29:53.546619 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:29:53.546873 (+ 254us) client_negotiation.cc:769] Sending connection context
0619 12:29:53.547058 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 12:29:53.547389 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":151,"threads_started":1}
I20260619 12:29:53.549592 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32999
I20260619 12:29:53.549835 1421 heartbeater.cc:464] Registering TS with master...
I20260619 12:29:53.550273 1421 heartbeater.cc:511] Master 127.0.98.62:32999 requested a full tablet report, sending...
I20260619 12:29:53.551268 962 ts_manager.cc:194] Registered new tserver with Master: 9c348efddd264229bec23e2eab0fae6a (127.0.98.3:44877)
I20260619 12:29:53.551834 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43021
I20260619 12:29:53.561923 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:29:53.574117 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:53.566530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58364 (local address 127.0.98.62:32999)
0619 12:29:53.566698 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:53.566703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:53.566864 (+ 161us) server_negotiation.cc:407] Connection header received
0619 12:29:53.567023 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:53.567027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:53.567079 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:53.567160 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:53.567903 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.568448 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.569354 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.569609 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.569723 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:53.571167 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:53.571171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:53.571174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:53.571207 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:53.572429 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:53.572837 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:53.572842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:53.572844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:53.572933 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:53.573218 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:53.573222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:53.573224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:53.573446 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:53.573603 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:53.573825 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 12:29:53.573950 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:29:53.579185 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:29:53.582319 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:29:53.597815 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:53.591635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55358 (local address 127.0.98.2:40887)
0619 12:29:53.592551 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:53.592559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:53.592583 (+ 24us) server_negotiation.cc:407] Connection header received
0619 12:29:53.592669 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:53.592674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:53.592889 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:53.593038 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:29:53.593412 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.594733 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.596351 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.597052 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.597197 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:53.597341 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:53.597540 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 12:29:53.597654 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":80,"threads_started":1}
I20260619 12:29:53.597820 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:53.591416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40887 (local address 127.0.0.1:55358)
0619 12:29:53.591832 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:53.591888 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:53.592026 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:53.593049 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:53.593056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:53.593080 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:29:53.593263 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:53.593272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.594853 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.594856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:53.596204 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:53.596213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.597202 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.597210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:53.597331 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:53.597371 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 12:29:53.597507 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 12:29:53.597643 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":86,"thread_start_us":142,"threads_started":1}
I20260619 12:29:53.598596 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:53.592315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38934 (local address 127.0.98.1:42129)
0619 12:29:53.592778 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:53.592786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:53.592952 (+ 166us) server_negotiation.cc:407] Connection header received
0619 12:29:53.593029 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:53.593034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:53.593248 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:53.593389 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:29:53.593630 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.594508 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.596332 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.597147 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.597277 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:53.597420 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:53.598314 (+ 894us) server_negotiation.cc:299] Negotiation successful
0619 12:29:53.598442 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":96,"threads_started":1}
I20260619 12:29:53.598781 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:53.592398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.0.98.3:44877)
0619 12:29:53.593398 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:53.593411 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:53.593434 (+ 23us) server_negotiation.cc:407] Connection header received
0619 12:29:53.593524 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:53.593533 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:53.593719 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:53.593879 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:29:53.594076 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.595208 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.596723 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.597515 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.598348 (+ 833us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:53.598469 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:53.598514 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 12:29:53.598592 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":119,"threads_started":1}
I20260619 12:29:53.598984 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:53.592205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44877 (local address 127.0.0.1:52434)
0619 12:29:53.592928 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:53.592940 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:53.593050 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:53.593869 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:53.593872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:53.593889 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:29:53.593982 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:53.593987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.595381 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.595384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:53.596586 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:53.596597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.597655 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.597659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:53.597753 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:53.597775 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:29:53.597834 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 12:29:53.597910 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":64,"thread_start_us":80,"threads_started":1}
I20260619 12:29:53.599150 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:53.591841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42129 (local address 127.0.0.1:38934)
0619 12:29:53.592755 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:53.592769 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:53.592855 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:53.593381 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:53.593384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:53.593400 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:29:53.593508 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:53.593523 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.594631 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.594635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:53.596203 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:53.596213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.597490 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.597497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:53.597598 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:53.597623 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 12:29:53.597694 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 12:29:53.597754 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":115,"thread_start_us":119,"threads_started":1}
I20260619 12:29:53.600807 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 948caa16c330406d80d87c9e2913b223 (DEFAULT_TABLE table=test-table [id=2dd2b18758d348ba8ad8c10405509c99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:29:53.601989 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948caa16c330406d80d87c9e2913b223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:29:53.602010 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 948caa16c330406d80d87c9e2913b223 (DEFAULT_TABLE table=test-table [id=2dd2b18758d348ba8ad8c10405509c99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:29:53.602995 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948caa16c330406d80d87c9e2913b223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:29:53.603662 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 948caa16c330406d80d87c9e2913b223 (DEFAULT_TABLE table=test-table [id=2dd2b18758d348ba8ad8c10405509c99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:29:53.604597 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948caa16c330406d80d87c9e2913b223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:29:53.610281 1440 tablet_bootstrap.cc:492] T 948caa16c330406d80d87c9e2913b223 P f3ba6e740ba4450a9ce1305d0d5618e0: Bootstrap starting.
I20260619 12:29:53.610697 1441 tablet_bootstrap.cc:492] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737: Bootstrap starting.
I20260619 12:29:53.611596 1442 tablet_bootstrap.cc:492] T 948caa16c330406d80d87c9e2913b223 P 9c348efddd264229bec23e2eab0fae6a: Bootstrap starting.
I20260619 12:29:53.613169 1441 tablet_bootstrap.cc:654] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737: Neither blocks nor log segments found. Creating new log.
I20260619 12:29:53.613483 1442 tablet_bootstrap.cc:654] T 948caa16c330406d80d87c9e2913b223 P 9c348efddd264229bec23e2eab0fae6a: Neither blocks nor log segments found. Creating new log.
I20260619 12:29:53.614068 1441 log.cc:824] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737: Log is configured to *not* fsync() on all Append() calls
I20260619 12:29:53.614562 1442 log.cc:824] T 948caa16c330406d80d87c9e2913b223 P 9c348efddd264229bec23e2eab0fae6a: Log is configured to *not* fsync() on all Append() calls
I20260619 12:29:53.616169 1441 tablet_bootstrap.cc:492] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737: No bootstrap required, opened a new log
I20260619 12:29:53.616277 1442 tablet_bootstrap.cc:492] T 948caa16c330406d80d87c9e2913b223 P 9c348efddd264229bec23e2eab0fae6a: No bootstrap required, opened a new log
I20260619 12:29:53.616376 1441 ts_tablet_manager.cc:1397] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:29:53.616475 1442 ts_tablet_manager.cc:1397] T 948caa16c330406d80d87c9e2913b223 P 9c348efddd264229bec23e2eab0fae6a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:29:53.616633 1440 tablet_bootstrap.cc:654] T 948caa16c330406d80d87c9e2913b223 P f3ba6e740ba4450a9ce1305d0d5618e0: Neither blocks nor log segments found. Creating new log.
I20260619 12:29:53.617532 1440 log.cc:824] T 948caa16c330406d80d87c9e2913b223 P f3ba6e740ba4450a9ce1305d0d5618e0: Log is configured to *not* fsync() on all Append() calls
I20260619 12:29:53.619529 1440 tablet_bootstrap.cc:492] T 948caa16c330406d80d87c9e2913b223 P f3ba6e740ba4450a9ce1305d0d5618e0: No bootstrap required, opened a new log
I20260619 12:29:53.619712 1440 ts_tablet_manager.cc:1397] T 948caa16c330406d80d87c9e2913b223 P f3ba6e740ba4450a9ce1305d0d5618e0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260619 12:29:53.619663 1442 raft_consensus.cc:348] T 948caa16c330406d80d87c9e2913b223 P 9c348efddd264229bec23e2eab0fae6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7fd94581434e3a83abf1216ee8c737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40887 } } peers { permanent_uuid: "9c348efddd264229bec23e2eab0fae6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44877 } } peers { permanent_uuid: "f3ba6e740ba4450a9ce1305d0d5618e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42129 } }
I20260619 12:29:53.619930 1442 raft_consensus.cc:374] T 948caa16c330406d80d87c9e2913b223 P 9c348efddd264229bec23e2eab0fae6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:29:53.619994 1442 raft_consensus.cc:729] T 948caa16c330406d80d87c9e2913b223 P 9c348efddd264229bec23e2eab0fae6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c348efddd264229bec23e2eab0fae6a, State: Initialized, Role: FOLLOWER
I20260619 12:29:53.620626 1442 consensus_queue.cc:260] T 948caa16c330406d80d87c9e2913b223 P 9c348efddd264229bec23e2eab0fae6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7fd94581434e3a83abf1216ee8c737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40887 } } peers { permanent_uuid: "9c348efddd264229bec23e2eab0fae6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44877 } } peers { permanent_uuid: "f3ba6e740ba4450a9ce1305d0d5618e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42129 } }
I20260619 12:29:53.622000 1442 ts_tablet_manager.cc:1428] T 948caa16c330406d80d87c9e2913b223 P 9c348efddd264229bec23e2eab0fae6a: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260619 12:29:53.622394 1421 heartbeater.cc:503] Master 127.0.98.62:32999 was elected leader, sending a full tablet report...
I20260619 12:29:53.622455 1441 raft_consensus.cc:348] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7fd94581434e3a83abf1216ee8c737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40887 } } peers { permanent_uuid: "9c348efddd264229bec23e2eab0fae6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44877 } } peers { permanent_uuid: "f3ba6e740ba4450a9ce1305d0d5618e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42129 } }
I20260619 12:29:53.622732 1441 raft_consensus.cc:374] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:29:53.622792 1441 raft_consensus.cc:729] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e7fd94581434e3a83abf1216ee8c737, State: Initialized, Role: FOLLOWER
I20260619 12:29:53.623241 1441 consensus_queue.cc:260] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7fd94581434e3a83abf1216ee8c737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40887 } } peers { permanent_uuid: "9c348efddd264229bec23e2eab0fae6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44877 } } peers { permanent_uuid: "f3ba6e740ba4450a9ce1305d0d5618e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42129 } }
I20260619 12:29:53.624228 1441 ts_tablet_manager.cc:1428] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260619 12:29:53.624742 1440 raft_consensus.cc:348] T 948caa16c330406d80d87c9e2913b223 P f3ba6e740ba4450a9ce1305d0d5618e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7fd94581434e3a83abf1216ee8c737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40887 } } peers { permanent_uuid: "9c348efddd264229bec23e2eab0fae6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44877 } } peers { permanent_uuid: "f3ba6e740ba4450a9ce1305d0d5618e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42129 } }
I20260619 12:29:53.625029 1440 raft_consensus.cc:374] T 948caa16c330406d80d87c9e2913b223 P f3ba6e740ba4450a9ce1305d0d5618e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:29:53.625118 1440 raft_consensus.cc:729] T 948caa16c330406d80d87c9e2913b223 P f3ba6e740ba4450a9ce1305d0d5618e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3ba6e740ba4450a9ce1305d0d5618e0, State: Initialized, Role: FOLLOWER
I20260619 12:29:53.625579 1440 consensus_queue.cc:260] T 948caa16c330406d80d87c9e2913b223 P f3ba6e740ba4450a9ce1305d0d5618e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7fd94581434e3a83abf1216ee8c737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40887 } } peers { permanent_uuid: "9c348efddd264229bec23e2eab0fae6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44877 } } peers { permanent_uuid: "f3ba6e740ba4450a9ce1305d0d5618e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42129 } }
I20260619 12:29:53.626546 1440 ts_tablet_manager.cc:1428] T 948caa16c330406d80d87c9e2913b223 P f3ba6e740ba4450a9ce1305d0d5618e0: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
W20260619 12:29:53.726737 1288 tablet.cc:2369] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:29:53.728477 1154 tablet.cc:2369] T 948caa16c330406d80d87c9e2913b223 P f3ba6e740ba4450a9ce1305d0d5618e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:29:53.760002 1447 raft_consensus.cc:482] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:29:53.760217 1447 raft_consensus.cc:504] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7fd94581434e3a83abf1216ee8c737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40887 } } peers { permanent_uuid: "9c348efddd264229bec23e2eab0fae6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44877 } } peers { permanent_uuid: "f3ba6e740ba4450a9ce1305d0d5618e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42129 } }
I20260619 12:29:53.761379 1447 leader_election.cc:288] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c348efddd264229bec23e2eab0fae6a (127.0.98.3:44877), f3ba6e740ba4450a9ce1305d0d5618e0 (127.0.98.1:42129)
I20260619 12:29:53.765574 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:53.761792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41565 (local address 127.0.98.1:42129)
0619 12:29:53.761930 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:53.761935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:53.762253 (+ 318us) server_negotiation.cc:407] Connection header received
0619 12:29:53.762432 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:53.762435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:53.762479 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:53.762580 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:29:53.762916 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.763431 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.764576 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.764988 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.765127 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:53.765203 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:53.765379 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 12:29:53.765450 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:29:53.765595 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:53.761689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42129 (local address 127.0.98.2:41565)
0619 12:29:53.762122 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:53.762135 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:53.762298 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:53.762590 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:53.762592 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:53.762607 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:29:53.762806 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:53.762811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.763540 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.763547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:53.764444 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:53.764453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.765141 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.765144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:53.765241 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:53.765264 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:29:53.765364 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 12:29:53.765436 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":140,"spinlock_wait_cycles":21888,"thread_start_us":112,"threads_started":1}
I20260619 12:29:53.765641 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:53.761835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52619 (local address 127.0.98.3:44877)
0619 12:29:53.761967 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:53.761972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:53.762214 (+ 242us) server_negotiation.cc:407] Connection header received
0619 12:29:53.762418 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:53.762423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:53.762477 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:53.762578 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:29:53.762907 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.763460 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.764614 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.765023 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.765211 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:53.765273 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:53.765454 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 12:29:53.765516 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:29:53.765671 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:53.761686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44877 (local address 127.0.98.2:52619)
0619 12:29:53.762127 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:53.762142 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:53.762282 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:53.762598 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:53.762601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:53.762616 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:29:53.762803 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:53.762808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.763602 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.763605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:53.764481 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:53.764489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.765218 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.765221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:53.765325 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:53.765345 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 12:29:53.765442 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 12:29:53.765510 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":1,"spinlock_wait_cycles":896,"thread_start_us":160,"threads_started":1}
I20260619 12:29:53.766252 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948caa16c330406d80d87c9e2913b223" candidate_uuid: "0e7fd94581434e3a83abf1216ee8c737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c348efddd264229bec23e2eab0fae6a" is_pre_election: true
I20260619 12:29:53.766264 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948caa16c330406d80d87c9e2913b223" candidate_uuid: "0e7fd94581434e3a83abf1216ee8c737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3ba6e740ba4450a9ce1305d0d5618e0" is_pre_election: true
I20260619 12:29:53.766547 1107 raft_consensus.cc:2454] T 948caa16c330406d80d87c9e2913b223 P f3ba6e740ba4450a9ce1305d0d5618e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e7fd94581434e3a83abf1216ee8c737 in term 0.
I20260619 12:29:53.766556 1375 raft_consensus.cc:2454] T 948caa16c330406d80d87c9e2913b223 P 9c348efddd264229bec23e2eab0fae6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e7fd94581434e3a83abf1216ee8c737 in term 0.
I20260619 12:29:53.767030 1174 leader_election.cc:302] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e7fd94581434e3a83abf1216ee8c737, 9c348efddd264229bec23e2eab0fae6a, f3ba6e740ba4450a9ce1305d0d5618e0; no voters:
I20260619 12:29:53.767323 1447 raft_consensus.cc:2781] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:29:53.767406 1447 raft_consensus.cc:482] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:29:53.767438 1447 raft_consensus.cc:3037] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737 [term 0 FOLLOWER]: Advancing to term 1
W20260619 12:29:53.783912 1422 tablet.cc:2369] T 948caa16c330406d80d87c9e2913b223 P 9c348efddd264229bec23e2eab0fae6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:29:53.829289 1447 raft_consensus.cc:504] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7fd94581434e3a83abf1216ee8c737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40887 } } peers { permanent_uuid: "9c348efddd264229bec23e2eab0fae6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44877 } } peers { permanent_uuid: "f3ba6e740ba4450a9ce1305d0d5618e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42129 } }
I20260619 12:29:53.829883 1447 leader_election.cc:288] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737 [CANDIDATE]: Term 1 election: Requested vote from peers 9c348efddd264229bec23e2eab0fae6a (127.0.98.3:44877), f3ba6e740ba4450a9ce1305d0d5618e0 (127.0.98.1:42129)
I20260619 12:29:53.830286 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948caa16c330406d80d87c9e2913b223" candidate_uuid: "0e7fd94581434e3a83abf1216ee8c737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c348efddd264229bec23e2eab0fae6a"
I20260619 12:29:53.830323 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948caa16c330406d80d87c9e2913b223" candidate_uuid: "0e7fd94581434e3a83abf1216ee8c737" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3ba6e740ba4450a9ce1305d0d5618e0"
I20260619 12:29:53.830444 1107 raft_consensus.cc:3037] T 948caa16c330406d80d87c9e2913b223 P f3ba6e740ba4450a9ce1305d0d5618e0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:29:53.830444 1375 raft_consensus.cc:3037] T 948caa16c330406d80d87c9e2913b223 P 9c348efddd264229bec23e2eab0fae6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:29:53.831859 1375 raft_consensus.cc:2454] T 948caa16c330406d80d87c9e2913b223 P 9c348efddd264229bec23e2eab0fae6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e7fd94581434e3a83abf1216ee8c737 in term 1.
I20260619 12:29:53.831859 1107 raft_consensus.cc:2454] T 948caa16c330406d80d87c9e2913b223 P f3ba6e740ba4450a9ce1305d0d5618e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e7fd94581434e3a83abf1216ee8c737 in term 1.
I20260619 12:29:53.832253 1174 leader_election.cc:302] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e7fd94581434e3a83abf1216ee8c737, 9c348efddd264229bec23e2eab0fae6a; no voters:
I20260619 12:29:53.832446 1447 raft_consensus.cc:2781] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:29:53.832680 1447 raft_consensus.cc:686] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737 [term 1 LEADER]: Becoming Leader. State: Replica: 0e7fd94581434e3a83abf1216ee8c737, State: Running, Role: LEADER
I20260619 12:29:53.833055 1447 consensus_queue.cc:237] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7fd94581434e3a83abf1216ee8c737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40887 } } peers { permanent_uuid: "9c348efddd264229bec23e2eab0fae6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44877 } } peers { permanent_uuid: "f3ba6e740ba4450a9ce1305d0d5618e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42129 } }
I20260619 12:29:53.837046 962 catalog_manager.cc:5495] T 948caa16c330406d80d87c9e2913b223 P 0e7fd94581434e3a83abf1216ee8c737 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e7fd94581434e3a83abf1216ee8c737 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0e7fd94581434e3a83abf1216ee8c737" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7fd94581434e3a83abf1216ee8c737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c348efddd264229bec23e2eab0fae6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3ba6e740ba4450a9ce1305d0d5618e0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42129 } health_report { overall_health: UNKNOWN } } }
Jun 19 12:29:53 dist-test-slave-fg3h krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:29:53.893638 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:53.884177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38944 (local address 127.0.98.1:42129)
0619 12:29:53.884333 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:53.884338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:53.884533 (+ 195us) server_negotiation.cc:407] Connection header received
0619 12:29:53.884698 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:53.884700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:53.884743 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:53.884816 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:53.885655 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.886170 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.887005 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.887234 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.887315 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:53.890031 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:53.890044 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:53.890083 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:53.890109 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:53.891817 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:53.892304 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:53.892310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:53.892311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:53.892392 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:53.892688 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:53.892691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:53.892693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:53.892937 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:53.893196 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:53.893344 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 12:29:53.893468 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 12:29:53 dist-test-slave-fg3h krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872193, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 12:29:53 dist-test-slave-fg3h krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872193, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:29:53.939828 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:53.930082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.0.98.1:42129)
0619 12:29:53.930232 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:53.930238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:53.930482 (+ 244us) server_negotiation.cc:407] Connection header received
0619 12:29:53.930642 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:53.930646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:53.930708 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:53.930805 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:53.931490 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.932360 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.933310 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:53.933530 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:53.933610 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:53.936772 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:53.936775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:53.936778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:53.936801 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:53.938279 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:53.938702 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:53.938705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:53.938706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:53.938783 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:53.939064 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:53.939066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:53.939067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:53.939243 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:53.939342 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:53.939592 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 12:29:53.939707 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 12:29:53 dist-test-slave-fg3h krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 12:29:54 dist-test-slave-fg3h krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:29:54.039449 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:54.030083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38962 (local address 127.0.98.1:42129)
0619 12:29:54.030250 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:54.030256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:54.030481 (+ 225us) server_negotiation.cc:407] Connection header received
0619 12:29:54.030698 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:54.030701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:54.030746 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:54.030850 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:54.031669 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:54.032164 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:54.033111 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:54.033331 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:54.033446 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:54.036252 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:54.036256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:54.036259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:54.036288 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:54.037840 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:54.038259 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:54.038264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:54.038266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:54.038344 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:54.038624 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:54.038627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:54.038629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:54.038798 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:54.038932 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:54.039165 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 12:29:54.039283 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 12:29:54.040757 1067 tablet_service.cc:2908] Rejecting scan request for tablet 948caa16c330406d80d87c9e2913b223: Uninitialized: clean time has not yet been initialized
I20260619 12:29:54.043030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 1024
I20260619 12:29:54.049170 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 1158
I20260619 12:29:54.055239 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 1292
I20260619 12:29:54.061220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 930
2026-06-19T12:29:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3509 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:29:54 dist-test-slave-fg3h krb5kdc[1465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:29:54 dist-test-slave-fg3h krb5kdc[1465](info): set up 2 sockets
Jun 19 12:29:54 dist-test-slave-fg3h krb5kdc[1465](info): commencing operation
krb5kdc: starting...
W20260619 12:29:55.484808 392 mini_kdc.cc:121] Time spent starting KDC: real 1.399s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:29:55 dist-test-slave-fg3h krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:29:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:29:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:29:55.608832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36133
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:29:55.718178 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:29:55.718487 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:29:55.718559 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:29:55.722550 1478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:29:55.722640 1478 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 12:29:55.722671 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:29:55.722712 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:29:55.722749 1478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:29:55.722784 1478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:29:55.728067 1478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36133
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:29:55.729171 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:29:55.730192 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:29:55.738786 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:55.738773 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:55.738814 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:55.739104 1478 server_base.cc:1034] running on GCE node
I20260619 12:29:55.739728 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:29:55.740988 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:29:55.742300 1478 hybrid_clock.cc:648] HybridClock initialized: now 1781872195742268 us; error 54 us; skew 500 ppm
Jun 19 12:29:55 dist-test-slave-fg3h krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:29:55.750674 1478 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:29:55.751681 1478 webserver.cc:460] Webserver started at http://127.0.98.62:42573/ using document root <none> and password file <none>
I20260619 12:29:55.752233 1478 fs_manager.cc:359] Metadata directory not provided
I20260619 12:29:55.752291 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:29:55.752460 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:29:55.754582 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "2134c1146ff34a63881d74649a15e2e4"
format_stamp: "Formatted at 2026-06-19 12:29:55 on dist-test-slave-fg3h"
server_key: "e024a34b4f2d1205d1f454296567c41f"
server_key_iv: "4fbfc4832320f0ce0100b2e3c0880c0d"
server_key_version: "encryptionkey@0"
I20260619 12:29:55.755117 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "2134c1146ff34a63881d74649a15e2e4"
format_stamp: "Formatted at 2026-06-19 12:29:55 on dist-test-slave-fg3h"
server_key: "e024a34b4f2d1205d1f454296567c41f"
server_key_iv: "4fbfc4832320f0ce0100b2e3c0880c0d"
server_key_version: "encryptionkey@0"
I20260619 12:29:55.758863 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 12:29:55.761368 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:55.762676 1478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:29:55.762825 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "2134c1146ff34a63881d74649a15e2e4"
format_stamp: "Formatted at 2026-06-19 12:29:55 on dist-test-slave-fg3h"
server_key: "e024a34b4f2d1205d1f454296567c41f"
server_key_iv: "4fbfc4832320f0ce0100b2e3c0880c0d"
server_key_version: "encryptionkey@0"
I20260619 12:29:55.762943 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:29:55.781847 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:29:55.785509 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:29:55.785723 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:29:55.793511 1478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36133
I20260619 12:29:55.793529 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36133 every 8 connection(s)
I20260619 12:29:55.794450 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:29:55.795322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 1478
I20260619 12:29:55.795413 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:29:55.795667 392 external_mini_cluster.cc:1428] Setting key ca0e89616507382ffbde7e034f4dee35
I20260619 12:29:55.797601 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:29:55.804109 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4: Bootstrap starting.
Jun 19 12:29:55 dist-test-slave-fg3h krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:29:55.806723 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4: Neither blocks nor log segments found. Creating new log.
I20260619 12:29:55.808032 1556 log.cc:824] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4: Log is configured to *not* fsync() on all Append() calls
I20260619 12:29:55.810331 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4: No bootstrap required, opened a new log
I20260619 12:29:55.810686 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:55.797053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.0.98.62:36133)
0619 12:29:55.797510 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:55.797529 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:55.797569 (+ 40us) server_negotiation.cc:407] Connection header received
0619 12:29:55.798393 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:55.798415 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:55.798816 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:55.799179 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:55.799813 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:55.801317 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:55.802191 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:55.802854 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:55.803030 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:55.805514 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:55.805521 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:55.805536 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:55.805565 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:55.807564 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:55.807973 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:55.807978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:55.807984 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:55.808132 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:55.808409 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:55.808414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:55.808416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:55.808866 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:55.809047 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:55.809352 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 12:29:55.809634 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":102,"threads_started":1}
I20260619 12:29:55.813807 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2134c1146ff34a63881d74649a15e2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36133 } }
I20260619 12:29:55.814018 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:29:55.814051 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2134c1146ff34a63881d74649a15e2e4, State: Initialized, Role: FOLLOWER
I20260619 12:29:55.814409 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2134c1146ff34a63881d74649a15e2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36133 } }
I20260619 12:29:55.814518 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:29:55.814556 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:29:55.814618 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:29:55.815497 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2134c1146ff34a63881d74649a15e2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36133 } }
I20260619 12:29:55.815846 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2134c1146ff34a63881d74649a15e2e4; no voters:
I20260619 12:29:55.816207 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:29:55.816330 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:29:55.816571 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4 [term 1 LEADER]: Becoming Leader. State: Replica: 2134c1146ff34a63881d74649a15e2e4, State: Running, Role: LEADER
I20260619 12:29:55.816954 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2134c1146ff34a63881d74649a15e2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36133 } }
I20260619 12:29:55.817414 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:29:55.818651 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2134c1146ff34a63881d74649a15e2e4. Latest consensus state: current_term: 1 leader_uuid: "2134c1146ff34a63881d74649a15e2e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2134c1146ff34a63881d74649a15e2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36133 } } }
I20260619 12:29:55.818709 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2134c1146ff34a63881d74649a15e2e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2134c1146ff34a63881d74649a15e2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36133 } } }
I20260619 12:29:55.818782 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4 [sys.catalog]: This master's current role is: LEADER
I20260619 12:29:55.818812 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4 [sys.catalog]: This master's current role is: LEADER
I20260619 12:29:55.819160 1571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:29:55.822554 1571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:29:55.828135 1571 catalog_manager.cc:1347] Generated new cluster ID: 257d847382314fc4944e1a0ae768a2a2
I20260619 12:29:55.828212 1571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:29:55.877755 1571 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:29:55.879046 1571 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:29:55.887385 1571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4: Generated new TSK 0
I20260619 12:29:55.888181 1571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:29:55.922012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36133
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:29:56.036345 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:29:56.036669 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:29:56.036756 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:29:56.041181 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:29:56.041276 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:29:56.041345 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:29:56.041416 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:29:56.047353 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:29:56.048441 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:29:56.049536 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:29:56.059065 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:56.059062 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:56.059168 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:56.059307 1582 server_base.cc:1034] running on GCE node
I20260619 12:29:56.059717 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:29:56.060381 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:29:56.061594 1582 hybrid_clock.cc:648] HybridClock initialized: now 1781872196061561 us; error 50 us; skew 500 ppm
Jun 19 12:29:56 dist-test-slave-fg3h krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:29:56.064455 1582 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:29:56.065331 1582 webserver.cc:460] Webserver started at http://127.0.98.1:37315/ using document root <none> and password file <none>
I20260619 12:29:56.065840 1582 fs_manager.cc:359] Metadata directory not provided
I20260619 12:29:56.065896 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:29:56.066251 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:29:56.068442 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "c70eab6385474555bc0ccfdd8a4bff16"
format_stamp: "Formatted at 2026-06-19 12:29:56 on dist-test-slave-fg3h"
server_key: "0bc00aaf15b9eee9038c383d2db4864d"
server_key_iv: "294f9ab7a53fe0d28eea473903752f84"
server_key_version: "encryptionkey@0"
I20260619 12:29:56.068945 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c70eab6385474555bc0ccfdd8a4bff16"
format_stamp: "Formatted at 2026-06-19 12:29:56 on dist-test-slave-fg3h"
server_key: "0bc00aaf15b9eee9038c383d2db4864d"
server_key_iv: "294f9ab7a53fe0d28eea473903752f84"
server_key_version: "encryptionkey@0"
I20260619 12:29:56.072736 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260619 12:29:56.075385 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:56.076676 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 12:29:56.076778 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "c70eab6385474555bc0ccfdd8a4bff16"
format_stamp: "Formatted at 2026-06-19 12:29:56 on dist-test-slave-fg3h"
server_key: "0bc00aaf15b9eee9038c383d2db4864d"
server_key_iv: "294f9ab7a53fe0d28eea473903752f84"
server_key_version: "encryptionkey@0"
I20260619 12:29:56.076865 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:29:56.086992 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:29:56.090327 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:29:56.090478 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:29:56.091792 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:29:56.091843 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:56.091883 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:29:56.091948 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:29:56 dist-test-slave-fg3h krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:29:56.102787 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38453
I20260619 12:29:56.102854 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38453 every 8 connection(s)
I20260619 12:29:56.103776 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:29:56.106861 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.092797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35863 (local address 127.0.98.62:36133)
0619 12:29:56.092943 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:56.092947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:56.093999 (+ 1052us) server_negotiation.cc:407] Connection header received
0619 12:29:56.095175 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:56.095179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:56.095242 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:56.095335 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:56.096910 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.097848 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.099630 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.099889 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.100008 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.103046 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:56.103049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:56.103052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:56.103088 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:56.104693 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:56.105344 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:56.105347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:56.105349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:56.105440 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:56.105785 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:56.105788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:56.105790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:56.105978 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:56.106086 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:56.106555 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 12:29:56.106692 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:29:56.107575 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.093368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36133 (local address 127.0.98.1:35863)
0619 12:29:56.093847 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:56.093888 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:56.094918 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:56.095513 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:56.095521 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:56.096021 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:29:56.096722 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.096743 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.097998 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.098002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.099510 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.099523 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.100059 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.100062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.100241 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.100933 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:29:56.100960 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:29:56.102864 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:29:56.104822 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:56.104829 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:56.104843 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:56.105234 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:56.105530 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:56.105534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:56.105536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:56.105683 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:56.106082 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:29:56.106088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:29:56.106321 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 12:29:56.106497 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 12:29:56.106799 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":133,"threads_started":1}
I20260619 12:29:56.108804 1713 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36133
I20260619 12:29:56.108829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 1582
I20260619 12:29:56.108948 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:29:56.108966 1713 heartbeater.cc:464] Registering TS with master...
I20260619 12:29:56.109285 392 external_mini_cluster.cc:1428] Setting key 21ea20853f93c4c329a61217079eac67
I20260619 12:29:56.109573 1713 heartbeater.cc:511] Master 127.0.98.62:36133 requested a full tablet report, sending...
I20260619 12:29:56.111199 1510 ts_manager.cc:194] Registered new tserver with Master: c70eab6385474555bc0ccfdd8a4bff16 (127.0.98.1:38453)
I20260619 12:29:56.112491 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35863
I20260619 12:29:56.120822 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.111177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36133 (local address 127.0.98.1:55685)
0619 12:29:56.113295 (+ 2118us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:56.113312 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:56.113423 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:56.113730 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:56.113732 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:56.113902 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:29:56.113995 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.114000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.114996 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.114998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.115605 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.115613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.116297 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.116303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.116417 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.117002 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:29:56.117017 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:29:56.117423 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:29:56.119423 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:56.119425 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:56.119427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:56.119615 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:56.119914 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:56.119916 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:56.119918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:56.119987 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:56.120410 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:29:56.120412 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:29:56.120482 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 12:29:56.120560 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 12:29:56.120678 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2011,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:29:56.120949 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.111256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55685 (local address 127.0.98.62:36133)
0619 12:29:56.111383 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:56.111389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:56.113397 (+ 2008us) server_negotiation.cc:407] Connection header received
0619 12:29:56.113564 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:56.113567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:56.113614 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:56.113708 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:56.114107 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.114866 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.115732 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.116149 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.116271 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.117559 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:56.117566 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:56.117572 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:56.117603 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:56.119317 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:56.119732 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:56.119738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:56.119741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:56.119825 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:56.120078 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:56.120082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:56.120086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:56.120314 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:56.120431 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:56.120643 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 12:29:56.120763 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:29:56.139640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36133
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:29:56.249744 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:29:56.249989 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:29:56.250031 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:29:56.253942 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:29:56.253990 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:29:56.254027 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:29:56.254060 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:29:56.259032 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:29:56.259940 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:29:56.260857 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:29:56.270890 1722 server_base.cc:1034] running on GCE node
W20260619 12:29:56.270872 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:56.270877 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:56.270908 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:56.271574 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:29:56.272269 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:29:56.273459 1722 hybrid_clock.cc:648] HybridClock initialized: now 1781872196273452 us; error 31 us; skew 500 ppm
Jun 19 12:29:56 dist-test-slave-fg3h krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:29:56.276156 1722 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:29:56.276993 1722 webserver.cc:460] Webserver started at http://127.0.98.2:44185/ using document root <none> and password file <none>
I20260619 12:29:56.277568 1722 fs_manager.cc:359] Metadata directory not provided
I20260619 12:29:56.277678 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:29:56.277905 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:29:56.279879 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "1c9e35b6cde94c2b8d937f75135055c5"
format_stamp: "Formatted at 2026-06-19 12:29:56 on dist-test-slave-fg3h"
server_key: "42c1ae181b9ff0d9dd9e9518aabbf3fd"
server_key_iv: "da6047d0ebaed74c55fedad5ddf53999"
server_key_version: "encryptionkey@0"
I20260619 12:29:56.280495 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1c9e35b6cde94c2b8d937f75135055c5"
format_stamp: "Formatted at 2026-06-19 12:29:56 on dist-test-slave-fg3h"
server_key: "42c1ae181b9ff0d9dd9e9518aabbf3fd"
server_key_iv: "da6047d0ebaed74c55fedad5ddf53999"
server_key_version: "encryptionkey@0"
I20260619 12:29:56.284319 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:29:56.287024 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:56.288211 1722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 12:29:56.288342 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "1c9e35b6cde94c2b8d937f75135055c5"
format_stamp: "Formatted at 2026-06-19 12:29:56 on dist-test-slave-fg3h"
server_key: "42c1ae181b9ff0d9dd9e9518aabbf3fd"
server_key_iv: "da6047d0ebaed74c55fedad5ddf53999"
server_key_version: "encryptionkey@0"
I20260619 12:29:56.288451 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:29:56.305887 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:29:56.309105 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:29:56.309333 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:29:56.310953 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:29:56.311029 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:56.311092 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:29:56.311131 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:29:56 dist-test-slave-fg3h krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:29:56.322444 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35409
I20260619 12:29:56.323438 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:29:56.324227 1852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35409 every 8 connection(s)
I20260619 12:29:56.325143 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.311697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38043 (local address 127.0.98.62:36133)
0619 12:29:56.311864 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:56.311871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:56.312671 (+ 800us) server_negotiation.cc:407] Connection header received
0619 12:29:56.313814 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:56.313817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:56.313887 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:56.313997 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:56.315247 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.316033 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.317344 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.317682 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.317803 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.321345 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:56.321349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:56.321351 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:56.321377 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:56.322997 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:56.323507 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:56.323510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:56.323512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:56.323607 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:56.323977 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:56.323981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:56.323982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:56.324229 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:56.324324 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:56.324864 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 12:29:56.324972 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 12:29:56.325881 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.312066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36133 (local address 127.0.98.2:38043)
0619 12:29:56.312507 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:56.312556 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:56.313514 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:56.314193 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:56.314200 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:56.314630 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:29:56.315096 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.315109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.316193 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.316197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.317228 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.317239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.317849 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.317853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.318049 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.318923 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:29:56.318987 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:29:56.321073 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:29:56.323115 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:56.323121 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:56.323135 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:56.323394 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:56.323700 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:56.323704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:56.323706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:56.323845 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:56.324382 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:29:56.324388 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:29:56.324615 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 12:29:56.324769 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 12:29:56.325004 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":128,"threads_started":1}
I20260619 12:29:56.326020 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 1722
I20260619 12:29:56.326112 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:29:56.326360 392 external_mini_cluster.cc:1428] Setting key 68eb843231b5daf3f7b4bf328091d9d7
I20260619 12:29:56.327455 1853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36133
I20260619 12:29:56.327617 1853 heartbeater.cc:464] Registering TS with master...
I20260619 12:29:56.328198 1853 heartbeater.cc:511] Master 127.0.98.62:36133 requested a full tablet report, sending...
I20260619 12:29:56.329696 1509 ts_manager.cc:194] Registered new tserver with Master: 1c9e35b6cde94c2b8d937f75135055c5 (127.0.98.2:35409)
I20260619 12:29:56.330504 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38043
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:29:56.337232 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.329648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56723 (local address 127.0.98.62:36133)
0619 12:29:56.329763 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:56.329770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:56.330016 (+ 246us) server_negotiation.cc:407] Connection header received
0619 12:29:56.330158 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:56.330161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:56.330203 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:56.330290 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:56.330713 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.331360 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.332350 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.332596 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.332702 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.334018 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:56.334021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:56.334023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:56.334043 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:56.335491 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:56.335976 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:56.335979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:56.335980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:56.336058 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:56.336385 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:56.336387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:56.336388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:56.336572 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:56.336661 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:56.336958 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 12:29:56.337049 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 12:29:56.337291 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.329684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36133 (local address 127.0.98.2:56723)
0619 12:29:56.329933 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:56.329945 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:56.330055 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:56.330305 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:56.330307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:56.330499 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:29:56.330601 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.330607 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.331483 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.331486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.332229 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.332244 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.332708 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.332711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.332801 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.333343 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:29:56.333365 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:29:56.333886 (+ 521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:29:56.335603 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:56.335606 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:56.335608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:56.335865 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:56.336167 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:56.336173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:56.336178 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:56.336291 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:56.336674 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:29:56.336679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:29:56.336792 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 12:29:56.336912 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 12:29:56.337067 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":54,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:29:56.354301 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36133
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:29:56.465483 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:29:56.465744 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:29:56.465832 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:29:56.470099 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:29:56.470160 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:29:56.470237 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:29:56.470322 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:29:56.476152 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:29:56.477352 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:29:56.478371 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:29:56.487982 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:56.487993 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:56.488072 1862 server_base.cc:1034] running on GCE node
W20260619 12:29:56.487974 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:56.488708 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:29:56.489363 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:29:56.490525 1862 hybrid_clock.cc:648] HybridClock initialized: now 1781872196490492 us; error 45 us; skew 500 ppm
Jun 19 12:29:56 dist-test-slave-fg3h krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:29:56.493445 1862 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:29:56.494345 1862 webserver.cc:460] Webserver started at http://127.0.98.3:45963/ using document root <none> and password file <none>
I20260619 12:29:56.494921 1862 fs_manager.cc:359] Metadata directory not provided
I20260619 12:29:56.494983 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:29:56.495198 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:29:56.497339 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "87cb3cce18684b959e207917300f7b44"
format_stamp: "Formatted at 2026-06-19 12:29:56 on dist-test-slave-fg3h"
server_key: "82dac974bd7f000e2147092b729e982e"
server_key_iv: "febb985a24fe55b5d392359a392da446"
server_key_version: "encryptionkey@0"
I20260619 12:29:56.497895 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "87cb3cce18684b959e207917300f7b44"
format_stamp: "Formatted at 2026-06-19 12:29:56 on dist-test-slave-fg3h"
server_key: "82dac974bd7f000e2147092b729e982e"
server_key_iv: "febb985a24fe55b5d392359a392da446"
server_key_version: "encryptionkey@0"
I20260619 12:29:56.501770 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 12:29:56.504347 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:56.505725 1862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:29:56.505870 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "87cb3cce18684b959e207917300f7b44"
format_stamp: "Formatted at 2026-06-19 12:29:56 on dist-test-slave-fg3h"
server_key: "82dac974bd7f000e2147092b729e982e"
server_key_iv: "febb985a24fe55b5d392359a392da446"
server_key_version: "encryptionkey@0"
I20260619 12:29:56.505988 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:29:56.523101 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:29:56.526477 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:29:56.526693 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:29:56.528252 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:29:56.528326 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:56.528398 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:29:56.528457 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:29:56 dist-test-slave-fg3h krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:29:56.539592 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34155
I20260619 12:29:56.539927 1992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34155 every 8 connection(s)
I20260619 12:29:56.540504 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:29:56.540859 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 1862
I20260619 12:29:56.540984 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:29:56.541328 392 external_mini_cluster.cc:1428] Setting key a8f0e35e97552a240b6d230158b4b204
I20260619 12:29:56.542764 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.529291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59793 (local address 127.0.98.62:36133)
0619 12:29:56.529452 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:56.529457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:56.530422 (+ 965us) server_negotiation.cc:407] Connection header received
0619 12:29:56.531778 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:56.531782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:56.531856 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:56.531972 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:56.533273 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.533799 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.535203 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.535443 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.535555 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.538633 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:56.538637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:56.538639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:56.538661 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:56.540383 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:56.541034 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:56.541039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:56.541040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:56.541152 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:56.541542 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:56.541544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:56.541545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:56.541729 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:56.541853 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:56.542469 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 12:29:56.542599 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 12:29:56.543576 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.529714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36133 (local address 127.0.98.3:59793)
0619 12:29:56.530235 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:56.530296 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:56.531471 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:56.532200 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:56.532207 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:56.532516 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:29:56.533127 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.533150 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.533920 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.533924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.535062 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.535072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.535604 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.535608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.535789 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.536464 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:29:56.536483 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:29:56.538459 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:29:56.540501 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:56.540506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:56.540519 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:56.540907 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:56.541272 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:56.541276 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:56.541277 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:56.541442 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:56.541911 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:29:56.541919 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:29:56.542174 (+ 255us) client_negotiation.cc:769] Sending connection context
0619 12:29:56.542390 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 12:29:56.542660 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":106,"threads_started":1}
I20260619 12:29:56.544752 1993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36133
I20260619 12:29:56.544884 1993 heartbeater.cc:464] Registering TS with master...
I20260619 12:29:56.545452 1993 heartbeater.cc:511] Master 127.0.98.62:36133 requested a full tablet report, sending...
I20260619 12:29:56.546480 1510 ts_manager.cc:194] Registered new tserver with Master: 87cb3cce18684b959e207917300f7b44 (127.0.98.3:34155)
I20260619 12:29:56.547288 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59793
I20260619 12:29:56.553925 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.547019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36133 (local address 127.0.98.3:47765)
0619 12:29:56.547273 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:56.547301 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:56.547419 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:56.547720 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:56.547723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:56.547952 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:29:56.548059 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.548066 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.548833 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.548836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.549542 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.549551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.549953 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.549955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.550021 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.550555 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:29:56.550566 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:29:56.550879 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:29:56.552430 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:56.552433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:56.552434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:56.552612 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:56.552908 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:56.552910 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:56.552911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:56.552981 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:56.553452 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:29:56.553454 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:29:56.553543 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 12:29:56.553624 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 12:29:56.553728 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":90,"threads_started":1}
I20260619 12:29:56.553963 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.547076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47765 (local address 127.0.98.62:36133)
0619 12:29:56.547200 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:56.547204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:56.547378 (+ 174us) server_negotiation.cc:407] Connection header received
0619 12:29:56.547495 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:56.547498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:56.547547 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:56.547620 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:56.548175 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.548702 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.549652 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.549859 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.549937 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.550988 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:56.550991 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:56.550993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:56.551010 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:56.552322 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:56.552736 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:56.552739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:56.552741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:56.552826 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:56.553069 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:56.553071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:56.553072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:56.553367 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:56.553449 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:56.553733 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 12:29:56.553847 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:29:56.555375 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 12:29:56 dist-test-slave-fg3h krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872196, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 12:29:56 dist-test-slave-fg3h krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872196, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:29:56.582204 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.573462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54004 (local address 127.0.98.62:36133)
0619 12:29:56.573622 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:56.573626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:56.573641 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:29:56.573689 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:56.573692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:56.573735 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:56.573820 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:56.574577 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.575246 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.576074 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.576355 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.576441 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.578902 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:56.578905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:56.578907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:56.578933 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:56.580435 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:56.580868 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:56.580873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:56.580874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:56.580977 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:56.581309 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:56.581314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:56.581315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:56.581499 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:56.581633 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:56.581897 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 12:29:56.582015 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 12:29:56.585494 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:29:56.588557 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:29:56.602727 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.597144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35409 (local address 127.0.0.1:49926)
0619 12:29:56.597526 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:56.597582 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:56.597711 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:56.598341 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:56.598347 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:56.598368 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:29:56.598519 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.598527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.599916 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.599919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.601049 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.601057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.602192 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.602196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.602301 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.602335 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 12:29:56.602447 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 12:29:56.602570 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":1,"spinlock_wait_cycles":5504,"thread_start_us":89,"threads_started":1}
I20260619 12:29:56.602725 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.597295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49926 (local address 127.0.98.2:35409)
0619 12:29:56.597849 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:56.597864 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:56.597886 (+ 22us) server_negotiation.cc:407] Connection header received
0619 12:29:56.597962 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:56.597967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:56.598189 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:56.598336 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:29:56.598668 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.599803 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.601237 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.602053 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.602205 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.602374 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:56.602486 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 12:29:56.602576 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":93,"threads_started":1}
I20260619 12:29:56.603848 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.597154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34155 (local address 127.0.0.1:51266)
0619 12:29:56.597959 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:56.597973 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:56.598054 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:56.599369 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:56.599372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:56.599399 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:29:56.599493 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.599498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.600597 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.600603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.602114 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.602120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.603428 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.603431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.603522 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.603545 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:29:56.603633 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 12:29:56.603709 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":171,"thread_start_us":94,"threads_started":1}
I20260619 12:29:56.603912 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.598117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51266 (local address 127.0.98.3:34155)
0619 12:29:56.598864 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:56.598874 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:56.598896 (+ 22us) server_negotiation.cc:407] Connection header received
0619 12:29:56.598966 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:56.598971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:56.599205 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:56.599623 (+ 418us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:29:56.599738 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.600473 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.602253 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.603276 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.603391 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.603548 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:56.603671 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 12:29:56.603763 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":125,"threads_started":1}
I20260619 12:29:56.604218 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.597884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38453 (local address 127.0.0.1:58662)
0619 12:29:56.598636 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:56.598656 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:56.598762 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:56.599308 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:56.599311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:56.599328 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:29:56.599434 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.599441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.600636 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.600638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.602095 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.602120 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.603612 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.603619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.603730 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.603752 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:29:56.603821 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 12:29:56.603886 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":660,"thread_start_us":70,"threads_started":1}
I20260619 12:29:56.604976 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.598032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.0.98.1:38453)
0619 12:29:56.598341 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:56.598355 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:56.598851 (+ 496us) server_negotiation.cc:407] Connection header received
0619 12:29:56.598929 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:56.598934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:56.599172 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:56.599331 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:29:56.599549 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.600541 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.602253 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.603300 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.603391 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.603548 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:56.604486 (+ 938us) server_negotiation.cc:299] Negotiation successful
0619 12:29:56.604589 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":104,"threads_started":1}
I20260619 12:29:56.606177 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 18aa05d706b04d818d47364a3aeb5853 (DEFAULT_TABLE table=test-table [id=b70b270411b54e1eb9500945afad208b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:29:56.606426 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 18aa05d706b04d818d47364a3aeb5853 (DEFAULT_TABLE table=test-table [id=b70b270411b54e1eb9500945afad208b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:29:56.607168 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18aa05d706b04d818d47364a3aeb5853. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:29:56.607314 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18aa05d706b04d818d47364a3aeb5853. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:29:56.611770 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 18aa05d706b04d818d47364a3aeb5853 (DEFAULT_TABLE table=test-table [id=b70b270411b54e1eb9500945afad208b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:29:56.612847 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18aa05d706b04d818d47364a3aeb5853. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:29:56.614697 2016 tablet_bootstrap.cc:492] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5: Bootstrap starting.
I20260619 12:29:56.617303 2016 tablet_bootstrap.cc:654] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5: Neither blocks nor log segments found. Creating new log.
I20260619 12:29:56.618093 2016 log.cc:824] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5: Log is configured to *not* fsync() on all Append() calls
I20260619 12:29:56.619313 2019 tablet_bootstrap.cc:492] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16: Bootstrap starting.
I20260619 12:29:56.619910 2017 tablet_bootstrap.cc:492] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44: Bootstrap starting.
I20260619 12:29:56.620157 2016 tablet_bootstrap.cc:492] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5: No bootstrap required, opened a new log
I20260619 12:29:56.620347 2016 ts_tablet_manager.cc:1397] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260619 12:29:56.622092 2019 tablet_bootstrap.cc:654] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16: Neither blocks nor log segments found. Creating new log.
I20260619 12:29:56.622092 2017 tablet_bootstrap.cc:654] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44: Neither blocks nor log segments found. Creating new log.
I20260619 12:29:56.622960 2019 log.cc:824] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16: Log is configured to *not* fsync() on all Append() calls
I20260619 12:29:56.622960 2017 log.cc:824] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44: Log is configured to *not* fsync() on all Append() calls
I20260619 12:29:56.623549 2016 raft_consensus.cc:348] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } } peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } }
I20260619 12:29:56.623850 2016 raft_consensus.cc:374] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:29:56.623919 2016 raft_consensus.cc:729] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c9e35b6cde94c2b8d937f75135055c5, State: Initialized, Role: FOLLOWER
I20260619 12:29:56.624358 2016 consensus_queue.cc:260] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } } peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } }
I20260619 12:29:56.624996 1853 heartbeater.cc:503] Master 127.0.98.62:36133 was elected leader, sending a full tablet report...
I20260619 12:29:56.625128 2019 tablet_bootstrap.cc:492] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16: No bootstrap required, opened a new log
I20260619 12:29:56.625134 2017 tablet_bootstrap.cc:492] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44: No bootstrap required, opened a new log
I20260619 12:29:56.625309 2019 ts_tablet_manager.cc:1397] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:29:56.625308 2017 ts_tablet_manager.cc:1397] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:29:56.625388 2016 ts_tablet_manager.cc:1428] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5: Time spent starting tablet: real 0.005s user 0.005s sys 0.002s
I20260619 12:29:56.629063 2019 raft_consensus.cc:348] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } } peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } }
I20260619 12:29:56.629375 2019 raft_consensus.cc:374] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:29:56.629432 2019 raft_consensus.cc:729] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c70eab6385474555bc0ccfdd8a4bff16, State: Initialized, Role: FOLLOWER
I20260619 12:29:56.629573 2017 raft_consensus.cc:348] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } } peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } }
I20260619 12:29:56.629808 2017 raft_consensus.cc:374] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:29:56.629868 2017 raft_consensus.cc:729] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87cb3cce18684b959e207917300f7b44, State: Initialized, Role: FOLLOWER
I20260619 12:29:56.629886 2019 consensus_queue.cc:260] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } } peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } }
I20260619 12:29:56.630314 2017 consensus_queue.cc:260] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } } peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } }
I20260619 12:29:56.630488 1713 heartbeater.cc:503] Master 127.0.98.62:36133 was elected leader, sending a full tablet report...
I20260619 12:29:56.630884 2019 ts_tablet_manager.cc:1428] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260619 12:29:56.631219 2017 ts_tablet_manager.cc:1428] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:29:56.631529 1993 heartbeater.cc:503] Master 127.0.98.62:36133 was elected leader, sending a full tablet report...
I20260619 12:29:56.632455 2023 raft_consensus.cc:482] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:29:56.632588 2023 raft_consensus.cc:504] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } } peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } }
I20260619 12:29:56.633736 2023 leader_election.cc:288] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c9e35b6cde94c2b8d937f75135055c5 (127.0.98.2:35409), 87cb3cce18684b959e207917300f7b44 (127.0.98.3:34155)
I20260619 12:29:56.638916 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.634523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47679 (local address 127.0.98.2:35409)
0619 12:29:56.634652 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:56.634656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:56.634669 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:29:56.634722 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:56.634725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:56.634775 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:56.634874 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:29:56.635179 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.635887 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.637544 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.638155 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.638290 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.638343 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:56.638722 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 12:29:56.638792 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:29:56.638916 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.634017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39951 (local address 127.0.98.3:34155)
0619 12:29:56.634293 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:56.634298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:56.635232 (+ 934us) server_negotiation.cc:407] Connection header received
0619 12:29:56.635284 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:56.635286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:56.635331 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:56.635392 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:29:56.635674 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.636162 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.637544 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.638157 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.638279 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.638343 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:56.638722 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 12:29:56.638792 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260619 12:29:56.638972 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.633899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35409 (local address 127.0.98.1:47679)
0619 12:29:56.634229 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:56.634246 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:56.634343 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:56.634897 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:56.634900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:56.634916 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:29:56.635048 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.635055 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.636041 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.636043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.637364 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.637374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.638333 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.638336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.638530 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.638607 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 12:29:56.638686 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 12:29:56.638784 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":121,"thread_start_us":105,"threads_started":1}
I20260619 12:29:56.638971 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.633956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34155 (local address 127.0.98.1:39951)
0619 12:29:56.635089 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:56.635104 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:56.635192 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:56.635427 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:56.635430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:56.635447 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:29:56.635553 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.635559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.636263 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.636266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.637408 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.637418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.638310 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.638316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.638526 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.638606 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 12:29:56.638686 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 12:29:56.638784 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"mutex_wait_us":84,"thread_start_us":103,"threads_started":1}
I20260619 12:29:56.639626 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18aa05d706b04d818d47364a3aeb5853" candidate_uuid: "c70eab6385474555bc0ccfdd8a4bff16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87cb3cce18684b959e207917300f7b44" is_pre_election: true
I20260619 12:29:56.639626 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18aa05d706b04d818d47364a3aeb5853" candidate_uuid: "c70eab6385474555bc0ccfdd8a4bff16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c9e35b6cde94c2b8d937f75135055c5" is_pre_election: true
I20260619 12:29:56.639917 1807 raft_consensus.cc:2454] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c70eab6385474555bc0ccfdd8a4bff16 in term 0.
I20260619 12:29:56.639917 1947 raft_consensus.cc:2454] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c70eab6385474555bc0ccfdd8a4bff16 in term 0.
I20260619 12:29:56.640413 1599 leader_election.cc:302] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c9e35b6cde94c2b8d937f75135055c5, c70eab6385474555bc0ccfdd8a4bff16; no voters:
I20260619 12:29:56.640673 2023 raft_consensus.cc:2781] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:29:56.640770 2023 raft_consensus.cc:482] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:29:56.640839 2023 raft_consensus.cc:3037] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:29:56.641947 2023 raft_consensus.cc:504] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } } peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } }
I20260619 12:29:56.642306 2023 leader_election.cc:288] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [CANDIDATE]: Term 1 election: Requested vote from peers 1c9e35b6cde94c2b8d937f75135055c5 (127.0.98.2:35409), 87cb3cce18684b959e207917300f7b44 (127.0.98.3:34155)
I20260619 12:29:56.642663 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18aa05d706b04d818d47364a3aeb5853" candidate_uuid: "c70eab6385474555bc0ccfdd8a4bff16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87cb3cce18684b959e207917300f7b44"
I20260619 12:29:56.642668 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18aa05d706b04d818d47364a3aeb5853" candidate_uuid: "c70eab6385474555bc0ccfdd8a4bff16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c9e35b6cde94c2b8d937f75135055c5"
I20260619 12:29:56.642788 1947 raft_consensus.cc:3037] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:29:56.642788 1807 raft_consensus.cc:3037] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:29:56.644075 1947 raft_consensus.cc:2454] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c70eab6385474555bc0ccfdd8a4bff16 in term 1.
I20260619 12:29:56.644075 1807 raft_consensus.cc:2454] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c70eab6385474555bc0ccfdd8a4bff16 in term 1.
I20260619 12:29:56.644385 1599 leader_election.cc:302] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c9e35b6cde94c2b8d937f75135055c5, c70eab6385474555bc0ccfdd8a4bff16; no voters:
I20260619 12:29:56.644603 2023 raft_consensus.cc:2781] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:29:56.644865 2023 raft_consensus.cc:686] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [term 1 LEADER]: Becoming Leader. State: Replica: c70eab6385474555bc0ccfdd8a4bff16, State: Running, Role: LEADER
I20260619 12:29:56.645270 2023 consensus_queue.cc:237] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } } peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } }
I20260619 12:29:56.648486 1510 catalog_manager.cc:5495] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 reported cstate change: term changed from 0 to 1, leader changed from <none> to c70eab6385474555bc0ccfdd8a4bff16 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c70eab6385474555bc0ccfdd8a4bff16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } health_report { overall_health: HEALTHY } } }
I20260619 12:29:56.672320 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.668465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58668 (local address 127.0.98.1:38453)
0619 12:29:56.668607 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:56.668614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:56.668632 (+ 18us) server_negotiation.cc:407] Connection header received
0619 12:29:56.668906 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:56.668909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:56.668970 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:56.669091 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:29:56.669386 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.669897 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.670802 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.671030 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.671157 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.671501 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:29:56.671905 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:29:56.672034 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:56.672109 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 12:29:56.672166 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:29:56.678572 1947 raft_consensus.cc:1261] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44 [term 1 FOLLOWER]: Refusing update from remote peer c70eab6385474555bc0ccfdd8a4bff16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:29:56.678573 1807 raft_consensus.cc:1261] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5 [term 1 FOLLOWER]: Refusing update from remote peer c70eab6385474555bc0ccfdd8a4bff16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:29:56.679347 2027 consensus_queue.cc:1035] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:29:56.679544 2023 consensus_queue.cc:1035] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:29:56.686879 2030 mvcc.cc:205] Tried to move back new op lower bound from 7298548517586243584 to 7298548517462167552. Current Snapshot: MvccSnapshot[applied={T|T < 7298548517586243584}]
I20260619 12:29:56.689623 2031 mvcc.cc:205] Tried to move back new op lower bound from 7298548517586243584 to 7298548517462167552. Current Snapshot: MvccSnapshot[applied={T|T < 7298548517586243584}]
I20260619 12:29:56.699409 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:29:56.699609 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:29:56.702049 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4: Sending DeleteTablet for 3 replicas of tablet 18aa05d706b04d818d47364a3aeb5853
I20260619 12:29:56.702930 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 18aa05d706b04d818d47364a3aeb5853 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:29:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49926
I20260619 12:29:56.703052 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 18aa05d706b04d818d47364a3aeb5853 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:29:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51266
I20260619 12:29:56.703056 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 18aa05d706b04d818d47364a3aeb5853 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:29:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58662
I20260619 12:29:56.703503 2040 tablet_replica.cc:330] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5: stopping tablet replica
I20260619 12:29:56.703590 2042 tablet_replica.cc:330] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16: stopping tablet replica
I20260619 12:29:56.703879 2042 raft_consensus.cc:2229] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:29:56.703879 2040 raft_consensus.cc:2229] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:29:56.704270 2040 raft_consensus.cc:2258] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:29:56.704269 2042 raft_consensus.cc:2258] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:29:56.704347 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:29:56.704771 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:29:56.705778 2042 ts_tablet_manager.cc:1905] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:29:56.705997 2040 ts_tablet_manager.cc:1905] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:29:56.706329 2041 tablet_replica.cc:330] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44: stopping tablet replica
I20260619 12:29:56.706616 2041 raft_consensus.cc:2229] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:29:56.706918 2041 raft_consensus.cc:2258] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:29:56.708950 2042 ts_tablet_manager.cc:1918] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:29:56.709051 2042 log.cc:1196] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-0/wal/wals/18aa05d706b04d818d47364a3aeb5853
I20260619 12:29:56.709406 2042 ts_tablet_manager.cc:1939] T 18aa05d706b04d818d47364a3aeb5853 P c70eab6385474555bc0ccfdd8a4bff16: Deleting consensus metadata
I20260619 12:29:56.710371 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 988e361e1b7a4e49827ba09c377ffd6c (DEFAULT_TABLE table=test-table [id=55da5030560a4e92a4fcee99bb5699b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:29:56.710371 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 988e361e1b7a4e49827ba09c377ffd6c (DEFAULT_TABLE table=test-table [id=55da5030560a4e92a4fcee99bb5699b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:29:56.710613 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 988e361e1b7a4e49827ba09c377ffd6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:29:56.710613 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 988e361e1b7a4e49827ba09c377ffd6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:29:56.711216 2041 ts_tablet_manager.cc:1905] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:29:56.712757 2019 tablet_bootstrap.cc:492] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16: Bootstrap starting.
I20260619 12:29:56.712981 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 988e361e1b7a4e49827ba09c377ffd6c (DEFAULT_TABLE table=test-table [id=55da5030560a4e92a4fcee99bb5699b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:29:56.713167 2016 tablet_bootstrap.cc:492] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5: Bootstrap starting.
I20260619 12:29:56.713259 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 988e361e1b7a4e49827ba09c377ffd6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:29:56.713726 2019 tablet_bootstrap.cc:654] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16: Neither blocks nor log segments found. Creating new log.
I20260619 12:29:56.713814 2041 ts_tablet_manager.cc:1918] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:29:56.713909 2041 log.cc:1196] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/18aa05d706b04d818d47364a3aeb5853
I20260619 12:29:56.714112 2016 tablet_bootstrap.cc:654] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5: Neither blocks nor log segments found. Creating new log.
I20260619 12:29:56.714216 2041 ts_tablet_manager.cc:1939] T 18aa05d706b04d818d47364a3aeb5853 P 87cb3cce18684b959e207917300f7b44: Deleting consensus metadata
I20260619 12:29:56.715444 2017 tablet_bootstrap.cc:492] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44: Bootstrap starting.
I20260619 12:29:56.716339 2017 tablet_bootstrap.cc:654] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44: Neither blocks nor log segments found. Creating new log.
I20260619 12:29:56.717070 1494 catalog_manager.cc:4841] TS c70eab6385474555bc0ccfdd8a4bff16 (127.0.98.1:38453): tablet 18aa05d706b04d818d47364a3aeb5853 (table test-table [id=b70b270411b54e1eb9500945afad208b]) successfully deleted
I20260619 12:29:56.717243 1494 catalog_manager.cc:4841] TS 87cb3cce18684b959e207917300f7b44 (127.0.98.3:34155): tablet 18aa05d706b04d818d47364a3aeb5853 (table test-table [id=b70b270411b54e1eb9500945afad208b]) successfully deleted
I20260619 12:29:56.717592 2017 tablet_bootstrap.cc:492] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44: No bootstrap required, opened a new log
I20260619 12:29:56.717665 2017 ts_tablet_manager.cc:1397] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:29:56.717684 2040 ts_tablet_manager.cc:1918] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:29:56.717784 2040 log.cc:1196] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/18aa05d706b04d818d47364a3aeb5853
I20260619 12:29:56.718148 2017 raft_consensus.cc:348] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } }
I20260619 12:29:56.718288 2016 tablet_bootstrap.cc:492] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5: No bootstrap required, opened a new log
I20260619 12:29:56.718292 2017 raft_consensus.cc:374] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:29:56.718328 2017 raft_consensus.cc:729] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87cb3cce18684b959e207917300f7b44, State: Initialized, Role: FOLLOWER
I20260619 12:29:56.718364 2016 ts_tablet_manager.cc:1397] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 12:29:56.718477 2017 consensus_queue.cc:260] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } }
I20260619 12:29:56.718643 2019 tablet_bootstrap.cc:492] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16: No bootstrap required, opened a new log
I20260619 12:29:56.718739 2019 ts_tablet_manager.cc:1397] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 12:29:56.718753 2017 ts_tablet_manager.cc:1428] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:29:56.718840 2016 raft_consensus.cc:348] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } }
I20260619 12:29:56.718971 2016 raft_consensus.cc:374] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:29:56.718999 2016 raft_consensus.cc:729] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c9e35b6cde94c2b8d937f75135055c5, State: Initialized, Role: FOLLOWER
I20260619 12:29:56.719125 2016 consensus_queue.cc:260] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } }
I20260619 12:29:56.719185 2019 raft_consensus.cc:348] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } }
I20260619 12:29:56.719309 2019 raft_consensus.cc:374] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:29:56.719358 2019 raft_consensus.cc:729] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c70eab6385474555bc0ccfdd8a4bff16, State: Initialized, Role: FOLLOWER
I20260619 12:29:56.719393 2016 ts_tablet_manager.cc:1428] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:29:56.719483 2019 consensus_queue.cc:260] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } }
I20260619 12:29:56.719722 2019 ts_tablet_manager.cc:1428] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:29:56.722388 2040 ts_tablet_manager.cc:1939] T 18aa05d706b04d818d47364a3aeb5853 P 1c9e35b6cde94c2b8d937f75135055c5: Deleting consensus metadata
I20260619 12:29:56.723840 1495 catalog_manager.cc:4841] TS 1c9e35b6cde94c2b8d937f75135055c5 (127.0.98.2:35409): tablet 18aa05d706b04d818d47364a3aeb5853 (table test-table [id=b70b270411b54e1eb9500945afad208b]) successfully deleted
W20260619 12:29:56.791689 1994 tablet.cc:2369] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:29:56.824813 1854 tablet.cc:2369] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:29:56.855165 1714 tablet.cc:2369] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:29:56.935586 2035 raft_consensus.cc:482] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:29:56.935730 2035 raft_consensus.cc:504] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } }
I20260619 12:29:56.936125 2035 leader_election.cc:288] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c9e35b6cde94c2b8d937f75135055c5 (127.0.98.2:35409), 87cb3cce18684b959e207917300f7b44 (127.0.98.3:34155)
I20260619 12:29:56.936650 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988e361e1b7a4e49827ba09c377ffd6c" candidate_uuid: "c70eab6385474555bc0ccfdd8a4bff16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87cb3cce18684b959e207917300f7b44" is_pre_election: true
I20260619 12:29:56.936650 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988e361e1b7a4e49827ba09c377ffd6c" candidate_uuid: "c70eab6385474555bc0ccfdd8a4bff16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c9e35b6cde94c2b8d937f75135055c5" is_pre_election: true
I20260619 12:29:56.936811 1807 raft_consensus.cc:2454] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c70eab6385474555bc0ccfdd8a4bff16 in term 0.
I20260619 12:29:56.936812 1947 raft_consensus.cc:2454] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c70eab6385474555bc0ccfdd8a4bff16 in term 0.
I20260619 12:29:56.937165 1599 leader_election.cc:302] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c9e35b6cde94c2b8d937f75135055c5, c70eab6385474555bc0ccfdd8a4bff16; no voters:
I20260619 12:29:56.937358 2035 raft_consensus.cc:2781] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:29:56.937412 2035 raft_consensus.cc:482] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:29:56.937439 2035 raft_consensus.cc:3037] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:29:56.938428 2035 raft_consensus.cc:504] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } }
I20260619 12:29:56.938735 2035 leader_election.cc:288] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [CANDIDATE]: Term 1 election: Requested vote from peers 1c9e35b6cde94c2b8d937f75135055c5 (127.0.98.2:35409), 87cb3cce18684b959e207917300f7b44 (127.0.98.3:34155)
I20260619 12:29:56.939092 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988e361e1b7a4e49827ba09c377ffd6c" candidate_uuid: "c70eab6385474555bc0ccfdd8a4bff16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87cb3cce18684b959e207917300f7b44"
I20260619 12:29:56.939116 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "988e361e1b7a4e49827ba09c377ffd6c" candidate_uuid: "c70eab6385474555bc0ccfdd8a4bff16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c9e35b6cde94c2b8d937f75135055c5"
I20260619 12:29:56.939222 1947 raft_consensus.cc:3037] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:29:56.939239 1807 raft_consensus.cc:3037] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:29:56.940122 1807 raft_consensus.cc:2454] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c70eab6385474555bc0ccfdd8a4bff16 in term 1.
I20260619 12:29:56.940166 1947 raft_consensus.cc:2454] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c70eab6385474555bc0ccfdd8a4bff16 in term 1.
I20260619 12:29:56.940390 1599 leader_election.cc:302] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c9e35b6cde94c2b8d937f75135055c5, c70eab6385474555bc0ccfdd8a4bff16; no voters:
I20260619 12:29:56.940534 2035 raft_consensus.cc:2781] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:29:56.940623 2035 raft_consensus.cc:686] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [term 1 LEADER]: Becoming Leader. State: Replica: c70eab6385474555bc0ccfdd8a4bff16, State: Running, Role: LEADER
I20260619 12:29:56.940797 2035 consensus_queue.cc:237] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } }
I20260619 12:29:56.942445 1510 catalog_manager.cc:5495] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 reported cstate change: term changed from 0 to 1, leader changed from <none> to c70eab6385474555bc0ccfdd8a4bff16 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c70eab6385474555bc0ccfdd8a4bff16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } health_report { overall_health: UNKNOWN } } }
Jun 19 12:29:56 dist-test-slave-fg3h krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:29:56.996261 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.986915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36133 (local address 127.0.0.1:54014)
0619 12:29:56.987202 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:56.987214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:56.987352 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:56.987615 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:56.987618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:56.988224 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:29:56.988395 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.988408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.989581 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.989584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.990385 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:56.990396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.990850 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.990853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:56.990921 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.991602 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:29:56.991614 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:29:56.993128 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:29:56.994752 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:56.994760 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:56.994766 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:56.995002 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:56.995279 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:56.995282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:56.995283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:56.995362 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:56.995773 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:29:56.995780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:29:56.995886 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 12:29:56.995981 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 12:29:56.996111 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":88,"threads_started":1}
I20260619 12:29:56.996284 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:56.986953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54014 (local address 127.0.98.62:36133)
0619 12:29:56.987102 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:56.987107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:56.987339 (+ 232us) server_negotiation.cc:407] Connection header received
0619 12:29:56.987466 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:56.987469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:56.987515 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:56.987599 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:56.988531 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.989336 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.990518 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:56.990741 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:56.990847 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:56.993357 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:56.993360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:56.993362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:56.993383 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:56.994582 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:56.995097 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:56.995099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:56.995101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:56.995173 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:56.995468 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:56.995471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:56.995472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:56.995653 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:56.995742 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:56.996040 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 12:29:56.996176 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":58}
I20260619 12:29:57.000682 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54014:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 12:29:57.001438 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:29:57.006299 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 418212663b5c4162bd84bc611a81d250 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a2a186ea60664f5a9d24744040fd4b34]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:29:57.006299 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 418212663b5c4162bd84bc611a81d250 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a2a186ea60664f5a9d24744040fd4b34]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:29:57.006328 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 418212663b5c4162bd84bc611a81d250 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a2a186ea60664f5a9d24744040fd4b34]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:29:57.006697 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 418212663b5c4162bd84bc611a81d250. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:29:57.006702 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 418212663b5c4162bd84bc611a81d250. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:29:57.006695 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 418212663b5c4162bd84bc611a81d250. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:29:57.009294 2017 tablet_bootstrap.cc:492] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44: Bootstrap starting.
I20260619 12:29:57.009832 2016 tablet_bootstrap.cc:492] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5: Bootstrap starting.
I20260619 12:29:57.010371 2017 tablet_bootstrap.cc:654] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44: Neither blocks nor log segments found. Creating new log.
I20260619 12:29:57.011220 2016 tablet_bootstrap.cc:654] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5: Neither blocks nor log segments found. Creating new log.
I20260619 12:29:57.011818 2019 tablet_bootstrap.cc:492] T 418212663b5c4162bd84bc611a81d250 P c70eab6385474555bc0ccfdd8a4bff16: Bootstrap starting.
I20260619 12:29:57.012511 2016 tablet_bootstrap.cc:492] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5: No bootstrap required, opened a new log
I20260619 12:29:57.012611 2016 ts_tablet_manager.cc:1397] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:29:57.012645 2019 tablet_bootstrap.cc:654] T 418212663b5c4162bd84bc611a81d250 P c70eab6385474555bc0ccfdd8a4bff16: Neither blocks nor log segments found. Creating new log.
I20260619 12:29:57.013103 2016 raft_consensus.cc:348] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } }
I20260619 12:29:57.013273 2016 raft_consensus.cc:374] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:29:57.013319 2016 raft_consensus.cc:729] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c9e35b6cde94c2b8d937f75135055c5, State: Initialized, Role: FOLLOWER
I20260619 12:29:57.013486 2016 consensus_queue.cc:260] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } }
I20260619 12:29:57.013916 2019 tablet_bootstrap.cc:492] T 418212663b5c4162bd84bc611a81d250 P c70eab6385474555bc0ccfdd8a4bff16: No bootstrap required, opened a new log
I20260619 12:29:57.013829 2022 tablet_replica.cc:439] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } } }
I20260619 12:29:57.013981 2022 tablet_replica.cc:442] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:29:57.014000 2019 ts_tablet_manager.cc:1397] T 418212663b5c4162bd84bc611a81d250 P c70eab6385474555bc0ccfdd8a4bff16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 12:29:57.013839 2016 ts_tablet_manager.cc:1428] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:29:57.014242 2017 tablet_bootstrap.cc:492] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44: No bootstrap required, opened a new log
I20260619 12:29:57.014328 2017 ts_tablet_manager.cc:1397] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 12:29:57.014395 2019 raft_consensus.cc:348] T 418212663b5c4162bd84bc611a81d250 P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } }
I20260619 12:29:57.014503 2019 raft_consensus.cc:374] T 418212663b5c4162bd84bc611a81d250 P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:29:57.014533 2019 raft_consensus.cc:729] T 418212663b5c4162bd84bc611a81d250 P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c70eab6385474555bc0ccfdd8a4bff16, State: Initialized, Role: FOLLOWER
I20260619 12:29:57.014642 2019 consensus_queue.cc:260] T 418212663b5c4162bd84bc611a81d250 P c70eab6385474555bc0ccfdd8a4bff16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } }
I20260619 12:29:57.014772 2017 raft_consensus.cc:348] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } }
I20260619 12:29:57.014879 2017 raft_consensus.cc:374] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:29:57.014909 2019 ts_tablet_manager.cc:1428] T 418212663b5c4162bd84bc611a81d250 P c70eab6385474555bc0ccfdd8a4bff16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:29:57.014911 2017 raft_consensus.cc:729] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87cb3cce18684b959e207917300f7b44, State: Initialized, Role: FOLLOWER
I20260619 12:29:57.015057 2017 consensus_queue.cc:260] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } }
I20260619 12:29:57.015080 2035 tablet_replica.cc:439] T 418212663b5c4162bd84bc611a81d250 P c70eab6385474555bc0ccfdd8a4bff16: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } } }
I20260619 12:29:57.015193 2035 tablet_replica.cc:442] T 418212663b5c4162bd84bc611a81d250 P c70eab6385474555bc0ccfdd8a4bff16: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:29:57.015293 2024 tablet_replica.cc:439] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } } }
I20260619 12:29:57.015409 2024 tablet_replica.cc:442] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:29:57.015966 2017 ts_tablet_manager.cc:1428] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 12:29:57.223421 2022 raft_consensus.cc:482] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:29:57.223577 2022 raft_consensus.cc:504] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } }
I20260619 12:29:57.224828 2022 leader_election.cc:288] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c70eab6385474555bc0ccfdd8a4bff16 (127.0.98.1:38453), 87cb3cce18684b959e207917300f7b44 (127.0.98.3:34155)
I20260619 12:29:57.227834 2024 raft_consensus.cc:482] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:29:57.227965 2024 raft_consensus.cc:504] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } }
I20260619 12:29:57.229254 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:57.225068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38453 (local address 127.0.98.2:49065)
0619 12:29:57.225380 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:57.225404 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:57.225543 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:57.225827 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:57.225832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:57.225857 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:29:57.225976 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:57.225983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.226906 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.226911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:57.227941 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:57.227952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.228786 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.228790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:57.228893 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:57.228925 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 12:29:57.228997 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 12:29:57.229076 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":79,"threads_started":1}
I20260619 12:29:57.229333 2024 leader_election.cc:288] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c70eab6385474555bc0ccfdd8a4bff16 (127.0.98.1:38453), 1c9e35b6cde94c2b8d937f75135055c5 (127.0.98.2:35409)
I20260619 12:29:57.229483 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:57.225506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34155 (local address 127.0.98.2:46707)
0619 12:29:57.225759 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:57.225775 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:57.225839 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:57.226086 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:57.226092 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:57.226111 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:29:57.226199 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:57.226204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.226996 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.226999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:57.228136 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:57.228146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.229023 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.229027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:57.229154 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:57.229177 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:29:57.229253 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 12:29:57.229337 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":53,"threads_started":1}
I20260619 12:29:57.229663 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:57.225187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49065 (local address 127.0.98.1:38453)
0619 12:29:57.225474 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:57.225480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:57.225499 (+ 19us) server_negotiation.cc:407] Connection header received
0619 12:29:57.225656 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:57.225660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:57.225713 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:57.225819 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:29:57.226126 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.226774 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.228088 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.228659 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.228767 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:57.228847 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:57.229146 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 12:29:57.229223 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":73,"threads_started":1}
I20260619 12:29:57.229851 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:57.225617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46707 (local address 127.0.98.3:34155)
0619 12:29:57.225834 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:57.225838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:57.225852 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:29:57.225927 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:57.225930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:57.225973 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:57.226062 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:29:57.226346 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.226886 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.228243 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.228885 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.229008 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:57.229063 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:57.229628 (+ 565us) server_negotiation.cc:299] Negotiation successful
0619 12:29:57.229713 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
I20260619 12:29:57.230701 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "418212663b5c4162bd84bc611a81d250" candidate_uuid: "1c9e35b6cde94c2b8d937f75135055c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87cb3cce18684b959e207917300f7b44" is_pre_election: true
I20260619 12:29:57.230756 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "418212663b5c4162bd84bc611a81d250" candidate_uuid: "1c9e35b6cde94c2b8d937f75135055c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c70eab6385474555bc0ccfdd8a4bff16" is_pre_election: true
I20260619 12:29:57.230868 1947 raft_consensus.cc:2454] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c9e35b6cde94c2b8d937f75135055c5 in term 0.
I20260619 12:29:57.230983 1667 raft_consensus.cc:2454] T 418212663b5c4162bd84bc611a81d250 P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c9e35b6cde94c2b8d937f75135055c5 in term 0.
I20260619 12:29:57.231254 1738 leader_election.cc:302] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c9e35b6cde94c2b8d937f75135055c5, 87cb3cce18684b959e207917300f7b44; no voters:
I20260619 12:29:57.231554 2022 raft_consensus.cc:2781] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:29:57.231612 2022 raft_consensus.cc:482] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:29:57.231649 2022 raft_consensus.cc:3037] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:29:57.232934 2022 raft_consensus.cc:504] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } }
I20260619 12:29:57.233353 2022 leader_election.cc:288] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 [CANDIDATE]: Term 1 election: Requested vote from peers c70eab6385474555bc0ccfdd8a4bff16 (127.0.98.1:38453), 87cb3cce18684b959e207917300f7b44 (127.0.98.3:34155)
I20260619 12:29:57.233722 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "418212663b5c4162bd84bc611a81d250" candidate_uuid: "1c9e35b6cde94c2b8d937f75135055c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87cb3cce18684b959e207917300f7b44"
I20260619 12:29:57.233847 1947 raft_consensus.cc:3037] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:29:57.233886 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:57.229636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38453 (local address 127.0.98.3:59381)
0619 12:29:57.229993 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:57.230009 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:57.230098 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:57.230589 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:57.230593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:57.230614 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:29:57.230716 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:57.230722 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.231697 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.231699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:57.232609 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:57.232618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.233452 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.233456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:57.233567 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:57.233590 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:29:57.233666 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 12:29:57.233740 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":10,"thread_start_us":76,"threads_started":1}
I20260619 12:29:57.233922 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:57.230217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59381 (local address 127.0.98.1:38453)
0619 12:29:57.230310 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:57.230314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:57.230328 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:29:57.230366 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:57.230369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:57.230414 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:57.230487 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:29:57.230849 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.231565 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.232736 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.233222 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.233328 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:57.233391 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:57.233702 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 12:29:57.233800 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 12:29:57.234476 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "418212663b5c4162bd84bc611a81d250" candidate_uuid: "1c9e35b6cde94c2b8d937f75135055c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c70eab6385474555bc0ccfdd8a4bff16"
I20260619 12:29:57.234568 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:57.229977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35409 (local address 127.0.98.3:42771)
0619 12:29:57.230312 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:57.230327 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:57.230431 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:57.230705 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:57.230707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:57.230719 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:29:57.230989 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:57.230997 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.232010 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.232013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:57.233125 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:57.233134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.234185 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.234188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:57.234278 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:57.234300 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:29:57.234387 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 12:29:57.234453 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":74,"threads_started":1}
I20260619 12:29:57.234545 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "418212663b5c4162bd84bc611a81d250" candidate_uuid: "87cb3cce18684b959e207917300f7b44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c70eab6385474555bc0ccfdd8a4bff16" is_pre_election: true
I20260619 12:29:57.234593 1667 raft_consensus.cc:3037] T 418212663b5c4162bd84bc611a81d250 P c70eab6385474555bc0ccfdd8a4bff16 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:29:57.234999 1947 raft_consensus.cc:2454] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c9e35b6cde94c2b8d937f75135055c5 in term 1.
I20260619 12:29:57.235275 1738 leader_election.cc:302] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c9e35b6cde94c2b8d937f75135055c5, 87cb3cce18684b959e207917300f7b44; no voters:
I20260619 12:29:57.235455 2022 raft_consensus.cc:2781] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:29:57.235698 2022 raft_consensus.cc:686] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 [term 1 LEADER]: Becoming Leader. State: Replica: 1c9e35b6cde94c2b8d937f75135055c5, State: Running, Role: LEADER
I20260619 12:29:57.235757 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:57.230239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42771 (local address 127.0.98.2:35409)
0619 12:29:57.230430 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:57.230436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:57.230451 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:29:57.230500 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:57.230503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:57.230540 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:57.230617 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:29:57.231153 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.231885 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.233262 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.233895 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.234155 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:57.234200 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:57.235488 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0619 12:29:57.235569 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260619 12:29:57.235991 2022 consensus_queue.cc:237] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } }
I20260619 12:29:57.236110 1667 raft_consensus.cc:2454] T 418212663b5c4162bd84bc611a81d250 P c70eab6385474555bc0ccfdd8a4bff16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c9e35b6cde94c2b8d937f75135055c5 in term 1.
I20260619 12:29:57.236176 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "418212663b5c4162bd84bc611a81d250" candidate_uuid: "87cb3cce18684b959e207917300f7b44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c9e35b6cde94c2b8d937f75135055c5" is_pre_election: true
I20260619 12:29:57.236644 1879 leader_election.cc:302] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87cb3cce18684b959e207917300f7b44; no voters: 1c9e35b6cde94c2b8d937f75135055c5, c70eab6385474555bc0ccfdd8a4bff16
I20260619 12:29:57.236927 2024 raft_consensus.cc:2726] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 12:29:57.237370 2063 tablet_replica.cc:439] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5: TxnStatusTablet state changed. Reason: New leader 1c9e35b6cde94c2b8d937f75135055c5. Latest consensus state: current_term: 1 leader_uuid: "1c9e35b6cde94c2b8d937f75135055c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } } }
I20260619 12:29:57.237483 2063 tablet_replica.cc:442] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:29:57.237759 2065 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 12:29:57.237939 2065 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 12:29:57.238420 1508 catalog_manager.cc:5495] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c9e35b6cde94c2b8d937f75135055c5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1c9e35b6cde94c2b8d937f75135055c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } health_report { overall_health: UNKNOWN } } }
I20260619 12:29:57.306171 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:57.302400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49942 (local address 127.0.98.2:35409)
0619 12:29:57.302521 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:57.302528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:57.302544 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:29:57.302795 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:57.302797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:57.302857 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:57.302968 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:29:57.303223 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.303733 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.304618 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.304858 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.304974 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:57.305333 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:29:57.305762 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:29:57.305889 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:57.305967 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 12:29:57.306027 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:29:57.306171 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:57.302296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35409 (local address 127.0.0.1:49942)
0619 12:29:57.302474 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:57.302492 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:57.302610 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:57.302983 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:57.302986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:57.302995 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:29:57.303107 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:57.303113 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.303859 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.303863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:57.304488 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:57.304497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.304976 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.304980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:57.305101 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:57.305152 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:29:57.305873 (+ 721us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:29:57.305888 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 12:29:57.305957 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 12:29:57.306059 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 12:29:57.309437 1947 raft_consensus.cc:1261] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44 [term 1 FOLLOWER]: Refusing update from remote peer 1c9e35b6cde94c2b8d937f75135055c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:29:57.309479 1667 raft_consensus.cc:1261] T 418212663b5c4162bd84bc611a81d250 P c70eab6385474555bc0ccfdd8a4bff16 [term 1 FOLLOWER]: Refusing update from remote peer 1c9e35b6cde94c2b8d937f75135055c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:29:57.309973 2022 consensus_queue.cc:1035] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:29:57.310143 2063 consensus_queue.cc:1035] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:29:57.311686 2024 tablet_replica.cc:439] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44: TxnStatusTablet state changed. Reason: New leader 1c9e35b6cde94c2b8d937f75135055c5. Latest consensus state: current_term: 1 leader_uuid: "1c9e35b6cde94c2b8d937f75135055c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } } }
I20260619 12:29:57.311822 2024 tablet_replica.cc:442] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:29:57.312088 2035 tablet_replica.cc:439] T 418212663b5c4162bd84bc611a81d250 P c70eab6385474555bc0ccfdd8a4bff16: TxnStatusTablet state changed. Reason: New leader 1c9e35b6cde94c2b8d937f75135055c5. Latest consensus state: current_term: 1 leader_uuid: "1c9e35b6cde94c2b8d937f75135055c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } } }
I20260619 12:29:57.312201 2035 tablet_replica.cc:442] T 418212663b5c4162bd84bc611a81d250 P c70eab6385474555bc0ccfdd8a4bff16: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:29:57.314471 2022 tablet_replica.cc:439] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c9e35b6cde94c2b8d937f75135055c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } } }
I20260619 12:29:57.314589 2022 tablet_replica.cc:442] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:29:57.315054 2035 tablet_replica.cc:439] T 418212663b5c4162bd84bc611a81d250 P c70eab6385474555bc0ccfdd8a4bff16: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c9e35b6cde94c2b8d937f75135055c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } } }
I20260619 12:29:57.315160 2035 tablet_replica.cc:442] T 418212663b5c4162bd84bc611a81d250 P c70eab6385474555bc0ccfdd8a4bff16: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:29:57.318117 2063 tablet_replica.cc:439] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c9e35b6cde94c2b8d937f75135055c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } } }
I20260619 12:29:57.318236 2063 tablet_replica.cc:442] T 418212663b5c4162bd84bc611a81d250 P 1c9e35b6cde94c2b8d937f75135055c5: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:29:57.319661 2069 mvcc.cc:205] Tried to move back new op lower bound from 7298548520173957120 to 7298548519880957952. Current Snapshot: MvccSnapshot[applied={T|T < 7298548520173957120}]
I20260619 12:29:57.319870 2024 tablet_replica.cc:439] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c9e35b6cde94c2b8d937f75135055c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70eab6385474555bc0ccfdd8a4bff16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38453 } } peers { permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 } } peers { permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 } } }
I20260619 12:29:57.319980 2024 tablet_replica.cc:442] T 418212663b5c4162bd84bc611a81d250 P 87cb3cce18684b959e207917300f7b44: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:29:57.333290 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:57.328898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35647 (local address 127.0.98.2:35409)
0619 12:29:57.329021 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:57.329026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:57.329314 (+ 288us) server_negotiation.cc:407] Connection header received
0619 12:29:57.329434 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:57.329437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:57.329492 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:57.329583 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:29:57.329842 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.330721 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.331985 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.332373 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.332487 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:57.332787 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:29:57.332877 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:29:57.332955 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:57.333060 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 12:29:57.333146 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 12:29:57.333323 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:57.328802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35409 (local address 127.0.98.1:35647)
0619 12:29:57.329208 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:57.329224 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:57.329344 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:57.329604 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:57.329608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:57.329617 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:29:57.329725 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:57.329731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.330846 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.330849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:57.331855 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:57.331866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.332498 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.332502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:57.332591 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:57.332646 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:29:57.332972 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:29:57.332978 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 12:29:57.333039 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 12:29:57.333153 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":99,"threads_started":1}
I20260619 12:29:57.341260 1947 raft_consensus.cc:1261] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44 [term 1 FOLLOWER]: Refusing update from remote peer c70eab6385474555bc0ccfdd8a4bff16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:29:57.341254 1807 raft_consensus.cc:1261] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5 [term 1 FOLLOWER]: Refusing update from remote peer c70eab6385474555bc0ccfdd8a4bff16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:29:57.341696 2035 consensus_queue.cc:1035] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87cb3cce18684b959e207917300f7b44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:29:57.341900 2027 consensus_queue.cc:1035] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c9e35b6cde94c2b8d937f75135055c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:29:57.367225 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:57.362038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56179 (local address 127.0.98.1:38453)
0619 12:29:57.362760 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:57.362769 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:57.362851 (+ 82us) server_negotiation.cc:407] Connection header received
0619 12:29:57.363016 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:57.363019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:57.363065 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:57.363149 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:29:57.363424 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.363990 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.365510 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.365729 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.365807 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:57.366715 (+ 908us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:29:57.366791 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:29:57.366870 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:57.367015 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 12:29:57.367076 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260619 12:29:57.367268 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:57.361961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38453 (local address 127.0.98.2:56179)
0619 12:29:57.362737 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:57.362793 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:57.362902 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:57.363170 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:57.363173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:57.363181 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:29:57.363316 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:57.363321 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.364141 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.364145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:57.365371 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:57.365388 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.366320 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.366328 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:57.366472 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:57.366531 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:29:57.366899 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:29:57.366907 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 12:29:57.366982 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 12:29:57.367090 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":102,"threads_started":1}
I20260619 12:29:57.390326 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:29:57.390533 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:29:57.392753 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2134c1146ff34a63881d74649a15e2e4: Sending DeleteTablet for 3 replicas of tablet 988e361e1b7a4e49827ba09c377ffd6c
I20260619 12:29:57.393394 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 988e361e1b7a4e49827ba09c377ffd6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:29:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49926
I20260619 12:29:57.393407 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 988e361e1b7a4e49827ba09c377ffd6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:29:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58662
I20260619 12:29:57.393746 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 988e361e1b7a4e49827ba09c377ffd6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:29:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51266
I20260619 12:29:57.393970 2088 tablet_replica.cc:330] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16: stopping tablet replica
I20260619 12:29:57.394047 2087 tablet_replica.cc:330] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5: stopping tablet replica
I20260619 12:29:57.394181 2087 raft_consensus.cc:2229] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:29:57.394240 2089 tablet_replica.cc:330] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44: stopping tablet replica
I20260619 12:29:57.394393 2089 raft_consensus.cc:2229] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:29:57.394431 2088 raft_consensus.cc:2229] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:29:57.394531 2089 raft_consensus.cc:2258] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:29:57.394691 2087 raft_consensus.cc:2258] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:29:57.394953 2088 raft_consensus.cc:2258] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:29:57.395282 2089 ts_tablet_manager.cc:1905] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:29:57.395879 2087 ts_tablet_manager.cc:1905] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:29:57.396543 2088 ts_tablet_manager.cc:1905] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:29:57.398591 2089 ts_tablet_manager.cc:1918] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:29:57.398689 2089 log.cc:1196] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/988e361e1b7a4e49827ba09c377ffd6c
I20260619 12:29:57.399013 2089 ts_tablet_manager.cc:1939] T 988e361e1b7a4e49827ba09c377ffd6c P 87cb3cce18684b959e207917300f7b44: Deleting consensus metadata
I20260619 12:29:57.399852 1494 catalog_manager.cc:4841] TS 87cb3cce18684b959e207917300f7b44 (127.0.98.3:34155): tablet 988e361e1b7a4e49827ba09c377ffd6c (table test-table [id=55da5030560a4e92a4fcee99bb5699b0]) successfully deleted
I20260619 12:29:57.401830 2087 ts_tablet_manager.cc:1918] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:29:57.401985 2087 log.cc:1196] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/988e361e1b7a4e49827ba09c377ffd6c
Jun 19 12:29:57 dist-test-slave-fg3h krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872196, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:29:57.402407 2087 ts_tablet_manager.cc:1939] T 988e361e1b7a4e49827ba09c377ffd6c P 1c9e35b6cde94c2b8d937f75135055c5: Deleting consensus metadata
I20260619 12:29:57.402772 2088 ts_tablet_manager.cc:1918] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:29:57.402858 2088 log.cc:1196] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781872188319340-392-0/minicluster-data/ts-0/wal/wals/988e361e1b7a4e49827ba09c377ffd6c
I20260619 12:29:57.403149 2088 ts_tablet_manager.cc:1939] T 988e361e1b7a4e49827ba09c377ffd6c P c70eab6385474555bc0ccfdd8a4bff16: Deleting consensus metadata
I20260619 12:29:57.403420 1495 catalog_manager.cc:4841] TS 1c9e35b6cde94c2b8d937f75135055c5 (127.0.98.2:35409): tablet 988e361e1b7a4e49827ba09c377ffd6c (table test-table [id=55da5030560a4e92a4fcee99bb5699b0]) successfully deleted
I20260619 12:29:57.403784 1494 catalog_manager.cc:4841] TS c70eab6385474555bc0ccfdd8a4bff16 (127.0.98.1:38453): tablet 988e361e1b7a4e49827ba09c377ffd6c (table test-table [id=55da5030560a4e92a4fcee99bb5699b0]) successfully deleted
I20260619 12:29:57.406823 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:57.395077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58674 (local address 127.0.98.1:38453)
0619 12:29:57.395186 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:57.395191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:57.395666 (+ 475us) server_negotiation.cc:407] Connection header received
0619 12:29:57.395854 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:57.395857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:57.395924 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:57.395984 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:57.397065 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.397826 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.398970 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.399278 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.399354 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:57.402718 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:57.402729 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:57.402745 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:57.402771 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:57.404714 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:57.405206 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:57.405213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:57.405214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:57.405304 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:57.405728 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:57.405735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:57.405738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:57.406008 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:57.406228 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:57.406533 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 12:29:57.406663 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 12:29:57.407272 1707 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58674
I20260619 12:29:57.418586 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:57.409133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54018 (local address 127.0.98.62:36133)
0619 12:29:57.409287 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:57.409298 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:57.409510 (+ 212us) server_negotiation.cc:407] Connection header received
0619 12:29:57.409658 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:57.409660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:57.409702 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:57.409765 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:57.410714 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.411237 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.412480 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:57.412712 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:57.412808 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:57.414867 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:57.414871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:57.414873 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:57.414899 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:57.416522 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:57.417057 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:57.417060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:57.417061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:57.417205 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:57.417538 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:57.417540 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:57.417541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:57.417722 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:57.417832 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:57.418129 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 12:29:57.418380 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 12:29:57.419155 1509 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54018
I20260619 12:29:57.420521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 1582
I20260619 12:29:57.429040 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 1722
I20260619 12:29:57.435086 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 1862
I20260619 12:29:57.440191 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 1478
2026-06-19T12:29:57Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3378 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:29:57 dist-test-slave-fg3h krb5kdc[2096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:29:57 dist-test-slave-fg3h krb5kdc[2096](info): set up 2 sockets
Jun 19 12:29:57 dist-test-slave-fg3h krb5kdc[2096](info): commencing operation
krb5kdc: starting...
W20260619 12:29:58.501327 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:29:58 dist-test-slave-fg3h krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:29:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:29:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:29:58.624994 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43019
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:29:58.737026 2109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:29:58.737334 2109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:29:58.737375 2109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:29:58.741596 2109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:29:58.741655 2109 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 12:29:58.741673 2109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:29:58.741693 2109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:29:58.741708 2109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:29:58.741770 2109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:29:58.747100 2109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43019
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:29:58.748229 2109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:29:58.749383 2109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:29:58.757876 2117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:58.757869 2114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:58.757876 2115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:58.758175 2109 server_base.cc:1034] running on GCE node
I20260619 12:29:58.758937 2109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:29:58.760124 2109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:29:58.761349 2109 hybrid_clock.cc:648] HybridClock initialized: now 1781872198761324 us; error 48 us; skew 500 ppm
Jun 19 12:29:58 dist-test-slave-fg3h krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:29:58.764460 2109 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:29:58.765496 2109 webserver.cc:460] Webserver started at http://127.0.98.62:42827/ using document root <none> and password file <none>
I20260619 12:29:58.766106 2109 fs_manager.cc:359] Metadata directory not provided
I20260619 12:29:58.766204 2109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:29:58.766453 2109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:29:58.768651 2109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "90d3391cea2c45c3b7ceb7f76e5c6abe"
format_stamp: "Formatted at 2026-06-19 12:29:58 on dist-test-slave-fg3h"
server_key: "57f74b547cbb4209ec026ba756a50d6e"
server_key_iv: "c8c02cf94b1a86bd26344f4d417b8391"
server_key_version: "encryptionkey@0"
I20260619 12:29:58.769309 2109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "90d3391cea2c45c3b7ceb7f76e5c6abe"
format_stamp: "Formatted at 2026-06-19 12:29:58 on dist-test-slave-fg3h"
server_key: "57f74b547cbb4209ec026ba756a50d6e"
server_key_iv: "c8c02cf94b1a86bd26344f4d417b8391"
server_key_version: "encryptionkey@0"
I20260619 12:29:58.773422 2109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 12:29:58.775846 2124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:58.777215 2109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:29:58.777350 2109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "90d3391cea2c45c3b7ceb7f76e5c6abe"
format_stamp: "Formatted at 2026-06-19 12:29:58 on dist-test-slave-fg3h"
server_key: "57f74b547cbb4209ec026ba756a50d6e"
server_key_iv: "c8c02cf94b1a86bd26344f4d417b8391"
server_key_version: "encryptionkey@0"
I20260619 12:29:58.777472 2109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:29:58.789597 2109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:29:58.792973 2109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:29:58.793208 2109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:29:58.801566 2109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43019
I20260619 12:29:58.801687 2186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43019 every 8 connection(s)
I20260619 12:29:58.802567 2109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:29:58.805361 2187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:29:58.810735 2187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe: Bootstrap starting.
I20260619 12:29:58.811405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 2109
I20260619 12:29:58.811494 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:29:58.811750 392 external_mini_cluster.cc:1428] Setting key 7ddd617e56916823c628418d7c8f2744
I20260619 12:29:58.813565 2187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe: Neither blocks nor log segments found. Creating new log.
I20260619 12:29:58.814451 2187 log.cc:824] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe: Log is configured to *not* fsync() on all Append() calls
I20260619 12:29:58.816993 2187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe: No bootstrap required, opened a new log
Jun 19 12:29:58 dist-test-slave-fg3h krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:29:58.821606 2187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d3391cea2c45c3b7ceb7f76e5c6abe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260619 12:29:58.821878 2187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:29:58.821955 2187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d3391cea2c45c3b7ceb7f76e5c6abe, State: Initialized, Role: FOLLOWER
I20260619 12:29:58.822435 2187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d3391cea2c45c3b7ceb7f76e5c6abe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260619 12:29:58.822619 2187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:29:58.822707 2187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:29:58.822805 2187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:29:58.823698 2187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d3391cea2c45c3b7ceb7f76e5c6abe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260619 12:29:58.824095 2187 leader_election.cc:302] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90d3391cea2c45c3b7ceb7f76e5c6abe; no voters:
I20260619 12:29:58.824455 2187 leader_election.cc:288] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:29:58.824555 2192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:29:58.824837 2192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe [term 1 LEADER]: Becoming Leader. State: Replica: 90d3391cea2c45c3b7ceb7f76e5c6abe, State: Running, Role: LEADER
I20260619 12:29:58.825245 2192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d3391cea2c45c3b7ceb7f76e5c6abe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260619 12:29:58.825870 2187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:29:58.826505 2192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90d3391cea2c45c3b7ceb7f76e5c6abe. Latest consensus state: current_term: 1 leader_uuid: "90d3391cea2c45c3b7ceb7f76e5c6abe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d3391cea2c45c3b7ceb7f76e5c6abe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } } }
I20260619 12:29:58.826740 2192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe [sys.catalog]: This master's current role is: LEADER
I20260619 12:29:58.826937 2193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90d3391cea2c45c3b7ceb7f76e5c6abe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90d3391cea2c45c3b7ceb7f76e5c6abe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } } }
I20260619 12:29:58.827824 2193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe [sys.catalog]: This master's current role is: LEADER
I20260619 12:29:58.827909 2201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:29:58.828006 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:58.813043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47640 (local address 127.0.98.62:43019)
0619 12:29:58.813540 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:58.813570 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:58.813616 (+ 46us) server_negotiation.cc:407] Connection header received
0619 12:29:58.814479 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:58.814502 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:58.814942 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:58.815286 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:58.816120 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:58.817205 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:58.818361 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:58.818845 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:58.818965 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:58.821823 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:58.821828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:58.821841 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:58.821868 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:58.823788 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:58.824333 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:58.824340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:58.824349 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:58.824495 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:58.824873 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:58.824876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:58.824878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:58.825426 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:58.825636 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:58.826255 (+ 619us) server_negotiation.cc:299] Negotiation successful
0619 12:29:58.826565 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":121,"threads_started":1}
I20260619 12:29:58.830853 2201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:29:58.836484 2201 catalog_manager.cc:1347] Generated new cluster ID: 5db794c07a3b48bbb74eb34271e663b6
I20260619 12:29:58.836571 2201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:29:58.858096 2201 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:29:58.859081 2201 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:29:58.867432 2201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe: Generated new TSK 0
I20260619 12:29:58.868134 2201 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:29:58.896492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:29:59.009196 2213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:29:59.009457 2213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:29:59.009537 2213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:29:59.013968 2213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:29:59.014034 2213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:29:59.014076 2213 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 12:29:59.014115 2213 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:29:59.014178 2213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:29:59.019750 2213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:29:59.020871 2213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:29:59.022034 2213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:29:59.032235 2218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:59.032230 2221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:59.032342 2213 server_base.cc:1034] running on GCE node
W20260619 12:29:59.032230 2219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:59.032843 2213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:29:59.033524 2213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:29:59.034775 2213 hybrid_clock.cc:648] HybridClock initialized: now 1781872199034743 us; error 52 us; skew 500 ppm
Jun 19 12:29:59 dist-test-slave-fg3h krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:29:59.037658 2213 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:29:59.038573 2213 webserver.cc:460] Webserver started at http://127.0.98.1:38325/ using document root <none> and password file <none>
I20260619 12:29:59.039155 2213 fs_manager.cc:359] Metadata directory not provided
I20260619 12:29:59.039273 2213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:29:59.039507 2213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:29:59.041822 2213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "3c6534fa12d84b19ab50ca02489beb17"
format_stamp: "Formatted at 2026-06-19 12:29:59 on dist-test-slave-fg3h"
server_key: "698af0bb7241e5f5b622223036988c65"
server_key_iv: "5d664a40fedf43c54ff04a484e55ed64"
server_key_version: "encryptionkey@0"
I20260619 12:29:59.042418 2213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3c6534fa12d84b19ab50ca02489beb17"
format_stamp: "Formatted at 2026-06-19 12:29:59 on dist-test-slave-fg3h"
server_key: "698af0bb7241e5f5b622223036988c65"
server_key_iv: "5d664a40fedf43c54ff04a484e55ed64"
server_key_version: "encryptionkey@0"
I20260619 12:29:59.046451 2213 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:29:59.049018 2228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:59.050524 2213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:29:59.050681 2213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "3c6534fa12d84b19ab50ca02489beb17"
format_stamp: "Formatted at 2026-06-19 12:29:59 on dist-test-slave-fg3h"
server_key: "698af0bb7241e5f5b622223036988c65"
server_key_iv: "5d664a40fedf43c54ff04a484e55ed64"
server_key_version: "encryptionkey@0"
I20260619 12:29:59.050815 2213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:29:59.066582 2213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:29:59.069599 2213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:29:59.069846 2213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:29:59.071303 2213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:29:59.071363 2213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:59.071417 2213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:29:59.071444 2213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:29:59 dist-test-slave-fg3h krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:29:59.082160 2213 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42197
I20260619 12:29:59.082248 2343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42197 every 8 connection(s)
I20260619 12:29:59.083274 2213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:29:59.086175 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.072146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39329 (local address 127.0.98.62:43019)
0619 12:29:59.072275 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:59.072280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:59.073284 (+ 1004us) server_negotiation.cc:407] Connection header received
0619 12:29:59.074430 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:59.074437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:59.074505 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:59.074639 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:59.076002 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.076900 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.078113 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.078544 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.078663 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.082138 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:59.082143 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:59.082148 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:59.082184 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:59.084098 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:59.084608 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:59.084612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:59.084614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:59.084700 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:59.085046 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:59.085050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:59.085052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:59.085290 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:59.085399 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:59.085882 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 12:29:59.086006 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 12:29:59.086910 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.072635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:39329)
0619 12:29:59.073116 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:59.073167 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:59.074130 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:59.074903 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:59.074912 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:59.075249 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:29:59.075835 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.075853 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.077041 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.077047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.077982 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.077991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.078741 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.078745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.078926 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.080029 (+ 1103us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:29:59.080045 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:29:59.081911 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:29:59.084225 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:59.084230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:59.084245 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:59.084496 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:59.084803 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:59.084806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:59.084808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:59.084946 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:59.085396 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:29:59.085403 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:29:59.085642 (+ 239us) client_negotiation.cc:769] Sending connection context
0619 12:29:59.085825 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 12:29:59.086132 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":128,"threads_started":1}
I20260619 12:29:59.088078 2344 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43019
I20260619 12:29:59.088335 2344 heartbeater.cc:464] Registering TS with master...
I20260619 12:29:59.088817 2344 heartbeater.cc:511] Master 127.0.98.62:43019 requested a full tablet report, sending...
I20260619 12:29:59.090564 2140 ts_manager.cc:194] Registered new tserver with Master: 3c6534fa12d84b19ab50ca02489beb17 (127.0.98.1:42197)
I20260619 12:29:59.091711 2140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39329
I20260619 12:29:59.093039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 2213
I20260619 12:29:59.093215 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:29:59.093549 392 external_mini_cluster.cc:1428] Setting key 43a0da91586bcfdf9c08081a1cb2a64f
I20260619 12:29:59.098639 2350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.090511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:54001)
0619 12:29:59.090911 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:59.090923 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:59.091039 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:59.091273 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:59.091276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:59.091473 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:29:59.091567 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.091572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.092410 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.092413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.093336 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.093346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.094083 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.094087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.094185 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.094743 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:29:59.094761 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:29:59.095215 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:29:59.097041 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:59.097047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:59.097050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:59.097358 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:59.097639 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:59.097644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:59.097647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:59.097752 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:59.098157 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:29:59.098160 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:29:59.098262 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 12:29:59.098372 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 12:29:59.098482 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":94,"threads_started":1}
I20260619 12:29:59.098896 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.090656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54001 (local address 127.0.98.62:43019)
0619 12:29:59.090771 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:59.090775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:59.090999 (+ 224us) server_negotiation.cc:407] Connection header received
0619 12:29:59.091118 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:59.091123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:59.091171 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:59.091257 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:59.091682 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.092306 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.093464 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.093838 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.093952 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.095359 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:59.095362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:59.095364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:59.095384 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:59.096923 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:59.097462 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:59.097465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:59.097466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:59.097542 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:59.097867 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:59.097870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:59.097871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:59.098053 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:59.098136 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:59.098471 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 12:29:59.098618 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:29:59.120591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:29:59.231138 2353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:29:59.231395 2353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:29:59.231443 2353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:29:59.235517 2353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:29:59.235579 2353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:29:59.235620 2353 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 12:29:59.235635 2353 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:29:59.235667 2353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:29:59.241488 2353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:29:59.242453 2353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:29:59.243470 2353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:29:59.252939 2358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:59.252967 2359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:59.252969 2361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:59.253153 2353 server_base.cc:1034] running on GCE node
I20260619 12:29:59.253641 2353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:29:59.254335 2353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:29:59.255501 2353 hybrid_clock.cc:648] HybridClock initialized: now 1781872199255469 us; error 45 us; skew 500 ppm
Jun 19 12:29:59 dist-test-slave-fg3h krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:29:59.258323 2353 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:29:59.259258 2353 webserver.cc:460] Webserver started at http://127.0.98.2:37919/ using document root <none> and password file <none>
I20260619 12:29:59.259825 2353 fs_manager.cc:359] Metadata directory not provided
I20260619 12:29:59.259917 2353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:29:59.260141 2353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:29:59.262184 2353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "eb6f7eb3e75b4d74abb614758f672e61"
format_stamp: "Formatted at 2026-06-19 12:29:59 on dist-test-slave-fg3h"
server_key: "274c2c82fd6fb4435608ac95d74b536a"
server_key_iv: "b8224bc3fe51166cd48ef9ca4d7aaa61"
server_key_version: "encryptionkey@0"
I20260619 12:29:59.262817 2353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eb6f7eb3e75b4d74abb614758f672e61"
format_stamp: "Formatted at 2026-06-19 12:29:59 on dist-test-slave-fg3h"
server_key: "274c2c82fd6fb4435608ac95d74b536a"
server_key_iv: "b8224bc3fe51166cd48ef9ca4d7aaa61"
server_key_version: "encryptionkey@0"
I20260619 12:29:59.266587 2353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:29:59.269145 2368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:59.270488 2353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 12:29:59.270680 2353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "eb6f7eb3e75b4d74abb614758f672e61"
format_stamp: "Formatted at 2026-06-19 12:29:59 on dist-test-slave-fg3h"
server_key: "274c2c82fd6fb4435608ac95d74b536a"
server_key_iv: "b8224bc3fe51166cd48ef9ca4d7aaa61"
server_key_version: "encryptionkey@0"
I20260619 12:29:59.270817 2353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:29:59.314286 2353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:29:59.317595 2353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:29:59.317828 2353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:29:59.319394 2353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:29:59.319530 2353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:59.319651 2353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:29:59.319689 2353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:29:59 dist-test-slave-fg3h krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:29:59.330907 2353 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42313
I20260619 12:29:59.330977 2483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42313 every 8 connection(s)
I20260619 12:29:59.331995 2353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:29:59.333702 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.320243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47159 (local address 127.0.98.62:43019)
0619 12:29:59.320390 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:59.320395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:59.321336 (+ 941us) server_negotiation.cc:407] Connection header received
0619 12:29:59.322305 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:59.322309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:59.322375 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:59.322457 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:59.323847 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.324576 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.325770 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.326100 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.326206 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.329516 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:59.329522 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:59.329525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:59.329556 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:59.331344 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:59.332008 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:59.332014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:59.332016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:59.332103 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:59.332492 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:59.332496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:59.332498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:59.332755 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:59.332861 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:59.333431 (+ 570us) server_negotiation.cc:299] Negotiation successful
0619 12:29:59.333538 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:29:59.334399 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.320706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:47159)
0619 12:29:59.321173 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:59.321222 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:59.322090 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:59.322766 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:59.322791 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:59.323155 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:29:59.323695 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.323708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.324698 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.324702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.325648 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.325657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.326423 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.326428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.326681 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.327443 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:29:59.327461 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:29:59.329291 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:29:59.331543 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:59.331549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:59.331563 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:59.331879 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:59.332185 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:59.332187 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:59.332188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:59.332332 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:59.332871 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:29:59.332880 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:29:59.333163 (+ 283us) client_negotiation.cc:769] Sending connection context
0619 12:29:59.333328 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 12:29:59.333571 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":129,"threads_started":1}
I20260619 12:29:59.335528 2484 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43019
I20260619 12:29:59.335765 2484 heartbeater.cc:464] Registering TS with master...
I20260619 12:29:59.336287 2484 heartbeater.cc:511] Master 127.0.98.62:43019 requested a full tablet report, sending...
I20260619 12:29:59.337163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 2353
I20260619 12:29:59.337281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:29:59.337409 2141 ts_manager.cc:194] Registered new tserver with Master: eb6f7eb3e75b4d74abb614758f672e61 (127.0.98.2:42313)
I20260619 12:29:59.337569 392 external_mini_cluster.cc:1428] Setting key 0d6606a8d7459e697c2286bffd617940
I20260619 12:29:59.338184 2141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47159
I20260619 12:29:59.346341 2490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.338129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:50843)
0619 12:29:59.338414 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:59.338430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:59.338548 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:59.338798 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:59.338800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:59.339026 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:29:59.339147 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.339155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.340117 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.340123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.340867 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.340876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.341492 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.341495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.341572 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.342249 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:29:59.342260 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:29:59.342569 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:29:59.344782 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:59.344785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:59.344786 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:59.344980 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:59.345346 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:59.345348 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:59.345349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:59.345425 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:59.345934 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:29:59.345937 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:29:59.346010 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 12:29:59.346096 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 12:29:59.346200 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":91,"threads_started":1}
I20260619 12:29:59.346493 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.338060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50843 (local address 127.0.98.62:43019)
0619 12:29:59.338177 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:59.338181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:59.338512 (+ 331us) server_negotiation.cc:407] Connection header received
0619 12:29:59.338653 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:59.338655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:59.338694 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:59.338774 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:59.339280 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.339971 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.340979 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.341378 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.341494 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.342762 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:59.342768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:59.342774 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:59.342805 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:59.344643 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:59.345129 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:59.345133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:59.345137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:59.345247 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:59.345525 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:59.345529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:59.345533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:59.345821 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:59.345936 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:59.346185 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 12:29:59.346319 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:29:59.364776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:29:59.473812 2493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:29:59.474069 2493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:29:59.474157 2493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:29:59.478228 2493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:29:59.478284 2493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:29:59.478324 2493 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 12:29:59.478338 2493 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:29:59.478369 2493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:29:59.483582 2493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:29:59.484565 2493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:29:59.485592 2493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:29:59.494904 2499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:29:59.494935 2498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:59.495002 2493 server_base.cc:1034] running on GCE node
W20260619 12:29:59.494915 2501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:29:59.495522 2493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:29:59.496138 2493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:29:59.497323 2493 hybrid_clock.cc:648] HybridClock initialized: now 1781872199497304 us; error 34 us; skew 500 ppm
Jun 19 12:29:59 dist-test-slave-fg3h krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:29:59.500003 2493 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:29:59.500883 2493 webserver.cc:460] Webserver started at http://127.0.98.3:43215/ using document root <none> and password file <none>
I20260619 12:29:59.501494 2493 fs_manager.cc:359] Metadata directory not provided
I20260619 12:29:59.501585 2493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:29:59.501818 2493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:29:59.503789 2493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "9db27fd360884e7e92427fe6a322c61f"
format_stamp: "Formatted at 2026-06-19 12:29:59 on dist-test-slave-fg3h"
server_key: "a50b76b145f23a90d83c806a993d6db8"
server_key_iv: "d868e641963aa5dfecb72e8eb1cba7b1"
server_key_version: "encryptionkey@0"
I20260619 12:29:59.504343 2493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9db27fd360884e7e92427fe6a322c61f"
format_stamp: "Formatted at 2026-06-19 12:29:59 on dist-test-slave-fg3h"
server_key: "a50b76b145f23a90d83c806a993d6db8"
server_key_iv: "d868e641963aa5dfecb72e8eb1cba7b1"
server_key_version: "encryptionkey@0"
I20260619 12:29:59.507983 2493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 12:29:59.510455 2508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:59.511713 2493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:29:59.511812 2493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "9db27fd360884e7e92427fe6a322c61f"
format_stamp: "Formatted at 2026-06-19 12:29:59 on dist-test-slave-fg3h"
server_key: "a50b76b145f23a90d83c806a993d6db8"
server_key_iv: "d868e641963aa5dfecb72e8eb1cba7b1"
server_key_version: "encryptionkey@0"
I20260619 12:29:59.511894 2493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:29:59.526523 2493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:29:59.530292 2493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:29:59.530443 2493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:29:59.531914 2493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:29:59.531965 2493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:29:59.532004 2493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:29:59.532020 2493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:29:59 dist-test-slave-fg3h krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:29:59.546887 2493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36531
I20260619 12:29:59.546901 2623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36531 every 8 connection(s)
I20260619 12:29:59.548112 2493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:29:59.550910 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.532717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50185 (local address 127.0.98.62:43019)
0619 12:29:59.532847 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:59.532851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:59.533950 (+ 1099us) server_negotiation.cc:407] Connection header received
0619 12:29:59.534996 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:59.534999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:59.535060 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:59.535160 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:59.536605 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.537211 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.539060 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.539449 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.539532 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.546955 (+ 7423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:59.546959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:59.546962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:59.547001 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:59.548683 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:59.549352 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:59.549356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:59.549358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:59.549445 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:59.549791 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:59.549794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:59.549796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:59.549990 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:59.550096 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:59.550588 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 12:29:59.550738 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:29:59.550992 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 2493
I20260619 12:29:59.551108 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:29:59.551447 392 external_mini_cluster.cc:1428] Setting key 8f215c9b6fd810baf216aa40b3174792
I20260619 12:29:59.551786 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.533306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:50185)
0619 12:29:59.533783 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:59.533849 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:59.534751 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:59.535386 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:59.535396 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:59.535777 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:29:59.536449 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.536466 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.537364 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.537368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.538904 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.538920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.539747 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.539754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.539980 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.542097 (+ 2117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:29:59.542147 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:29:59.546707 (+ 4560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:29:59.548815 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:59.548830 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:59.548844 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:59.549255 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:59.549544 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:59.549549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:59.549551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:59.549689 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:59.550093 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:29:59.550100 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:29:59.550355 (+ 255us) client_negotiation.cc:769] Sending connection context
0619 12:29:59.550534 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 12:29:59.550874 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":134,"threads_started":1}
I20260619 12:29:59.553143 2624 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43019
I20260619 12:29:59.553287 2624 heartbeater.cc:464] Registering TS with master...
I20260619 12:29:59.553853 2624 heartbeater.cc:511] Master 127.0.98.62:43019 requested a full tablet report, sending...
I20260619 12:29:59.555877 2140 ts_manager.cc:194] Registered new tserver with Master: 9db27fd360884e7e92427fe6a322c61f (127.0.98.3:36531)
I20260619 12:29:59.556721 2140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50185
I20260619 12:29:59.563951 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.555530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38731 (local address 127.0.98.62:43019)
0619 12:29:59.555653 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:59.555658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:59.555920 (+ 262us) server_negotiation.cc:407] Connection header received
0619 12:29:59.556049 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:59.556052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:59.556101 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:59.556164 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:59.556634 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.557365 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.558241 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.558571 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.558853 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.560215 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:59.560218 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:59.560220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:59.560241 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:59.561999 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:59.562491 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:59.562496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:59.562498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:59.562617 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:59.562977 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:59.562981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:59.562982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:59.563191 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:59.563293 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:59.563649 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 12:29:59.563778 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 12:29:59.563973 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.555538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:38731)
0619 12:29:59.555818 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:59.555834 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:59.555942 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:59.556205 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:59.556208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:59.556420 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:29:59.556534 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.556540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.557482 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.557488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.558052 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.558059 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.558881 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.558886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.559011 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.559548 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:29:59.559569 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:29:59.560074 (+ 505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:29:59.562091 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:59.562113 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:59.562118 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:59.562376 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:59.562756 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:29:59.562761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:29:59.562765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:29:59.562869 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:29:59.563311 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:29:59.563314 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:29:59.563441 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 12:29:59.563579 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 12:29:59.563729 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":99,"threads_started":1}
I20260619 12:29:59.567380 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 12:29:59 dist-test-slave-fg3h krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 12:29:59 dist-test-slave-fg3h krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:29:59.643918 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.634428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47654 (local address 127.0.98.62:43019)
0619 12:29:59.634560 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:59.634564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:59.634577 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:29:59.634619 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:59.634621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:59.634664 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:59.634740 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:29:59.635794 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.636503 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.637902 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.638117 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.638202 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.640760 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:29:59.640763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:29:59.640765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:29:59.640790 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:29:59.642313 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:59.642731 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:59.642733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:59.642735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:59.642821 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:29:59.643081 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:29:59.643083 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:29:59.643084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:29:59.643272 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:29:59.643377 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:59.643617 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 12:29:59.643720 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 12:29:59.647373 2140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:29:59.650545 2140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:29:59.665254 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.659483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42197 (local address 127.0.0.1:43324)
0619 12:29:59.659819 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:59.659872 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:59.660006 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:59.660955 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:59.660961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:59.660985 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:29:59.661165 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.661174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.662472 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.662475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.663492 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.663504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.664608 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.664612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.664764 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.664824 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 12:29:59.664944 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 12:29:59.665073 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":5,"thread_start_us":70,"threads_started":1}
I20260619 12:29:59.666164 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.660242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36531 (local address 127.0.0.1:59096)
0619 12:29:59.660815 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:59.661263 (+ 448us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:59.661369 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:59.661909 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:59.661912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:59.661940 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:29:59.662034 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.662040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.663405 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.663408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.664843 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.664853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.665777 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.665781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.665878 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.665900 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:29:59.665968 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 12:29:59.666037 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":229,"thread_start_us":91,"threads_started":1}
I20260619 12:29:59.666229 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.660639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59096 (local address 127.0.98.3:36531)
0619 12:29:59.661062 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:59.661073 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:59.661345 (+ 272us) server_negotiation.cc:407] Connection header received
0619 12:29:59.661473 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:59.661478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:59.661633 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:59.661798 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:29:59.662153 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.663256 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.664968 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.665651 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.665779 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.665953 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:59.666006 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 12:29:59.666092 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":116,"threads_started":1}
I20260619 12:29:59.666287 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.659658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42313 (local address 127.0.0.1:52692)
0619 12:29:59.660248 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:59.660261 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:59.660353 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:59.661765 (+ 1412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:59.661768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:59.661784 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:29:59.661883 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.661890 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.663405 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.663408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.664775 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.664785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.665843 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.665847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.665949 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.665971 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:29:59.666032 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 12:29:59.666108 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":151,"thread_start_us":97,"threads_started":1}
I20260619 12:29:59.666388 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.660271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52692 (local address 127.0.98.2:42313)
0619 12:29:59.661323 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:59.661335 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:59.661356 (+ 21us) server_negotiation.cc:407] Connection header received
0619 12:29:59.661424 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:59.661428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:59.661617 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:59.661764 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:29:59.661985 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.663257 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.664922 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.665743 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.665851 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.666011 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:59.666072 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 12:29:59.666151 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":950,"thread_start_us":109,"threads_started":1}
I20260619 12:29:59.666678 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.659627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.0.98.1:42197)
0619 12:29:59.660432 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:59.660439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:59.660465 (+ 26us) server_negotiation.cc:407] Connection header received
0619 12:29:59.660542 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:59.660547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:59.660757 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:59.660986 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:29:59.661303 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.662349 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.663629 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.664488 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.664609 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.664759 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:59.666441 (+ 1682us) server_negotiation.cc:299] Negotiation successful
0619 12:29:59.666556 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":82,"threads_started":1}
I20260619 12:29:59.669309 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 6e9eea2338fe480596de01b205e616fe (DEFAULT_TABLE table=test-table [id=bde3f81170c94919b3a9672027a072bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:29:59.669332 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 6e9eea2338fe480596de01b205e616fe (DEFAULT_TABLE table=test-table [id=bde3f81170c94919b3a9672027a072bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:29:59.669323 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 6e9eea2338fe480596de01b205e616fe (DEFAULT_TABLE table=test-table [id=bde3f81170c94919b3a9672027a072bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:29:59.670320 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e9eea2338fe480596de01b205e616fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:29:59.670331 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e9eea2338fe480596de01b205e616fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:29:59.670336 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e9eea2338fe480596de01b205e616fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:29:59.677397 2647 tablet_bootstrap.cc:492] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f: Bootstrap starting.
I20260619 12:29:59.677966 2648 tablet_bootstrap.cc:492] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17: Bootstrap starting.
I20260619 12:29:59.678399 2649 tablet_bootstrap.cc:492] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61: Bootstrap starting.
I20260619 12:29:59.679602 2647 tablet_bootstrap.cc:654] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f: Neither blocks nor log segments found. Creating new log.
I20260619 12:29:59.680033 2648 tablet_bootstrap.cc:654] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17: Neither blocks nor log segments found. Creating new log.
I20260619 12:29:59.680292 2649 tablet_bootstrap.cc:654] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61: Neither blocks nor log segments found. Creating new log.
I20260619 12:29:59.680467 2647 log.cc:824] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f: Log is configured to *not* fsync() on all Append() calls
I20260619 12:29:59.680864 2648 log.cc:824] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17: Log is configured to *not* fsync() on all Append() calls
I20260619 12:29:59.680974 2649 log.cc:824] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61: Log is configured to *not* fsync() on all Append() calls
I20260619 12:29:59.682791 2647 tablet_bootstrap.cc:492] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f: No bootstrap required, opened a new log
I20260619 12:29:59.682821 2649 tablet_bootstrap.cc:492] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61: No bootstrap required, opened a new log
I20260619 12:29:59.682826 2648 tablet_bootstrap.cc:492] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17: No bootstrap required, opened a new log
I20260619 12:29:59.682996 2648 ts_tablet_manager.cc:1397] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:29:59.682996 2649 ts_tablet_manager.cc:1397] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 12:29:59.683003 2647 ts_tablet_manager.cc:1397] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 12:29:59.685937 2647 raft_consensus.cc:348] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:29:59.686170 2647 raft_consensus.cc:374] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:29:59.686223 2647 raft_consensus.cc:729] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db27fd360884e7e92427fe6a322c61f, State: Initialized, Role: FOLLOWER
I20260619 12:29:59.686676 2647 consensus_queue.cc:260] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:29:59.686926 2649 raft_consensus.cc:348] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:29:59.687168 2649 raft_consensus.cc:374] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:29:59.687232 2649 raft_consensus.cc:729] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6f7eb3e75b4d74abb614758f672e61, State: Initialized, Role: FOLLOWER
I20260619 12:29:59.687405 2648 raft_consensus.cc:348] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:29:59.687626 2648 raft_consensus.cc:374] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:29:59.687682 2648 raft_consensus.cc:729] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c6534fa12d84b19ab50ca02489beb17, State: Initialized, Role: FOLLOWER
I20260619 12:29:59.687706 2647 ts_tablet_manager.cc:1428] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:29:59.687676 2649 consensus_queue.cc:260] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:29:59.688071 2624 heartbeater.cc:503] Master 127.0.98.62:43019 was elected leader, sending a full tablet report...
I20260619 12:29:59.688148 2648 consensus_queue.cc:260] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:29:59.688624 2649 ts_tablet_manager.cc:1428] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 12:29:59.688931 2484 heartbeater.cc:503] Master 127.0.98.62:43019 was elected leader, sending a full tablet report...
I20260619 12:29:59.689004 2648 ts_tablet_manager.cc:1428] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 12:29:59.689356 2344 heartbeater.cc:503] Master 127.0.98.62:43019 was elected leader, sending a full tablet report...
W20260619 12:29:59.799269 2625 tablet.cc:2369] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:29:59.833624 2485 tablet.cc:2369] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:29:59.834576 2345 tablet.cc:2369] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:29:59.929165 2653 raft_consensus.cc:482] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:29:59.929423 2653 raft_consensus.cc:504] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:29:59.930600 2653 leader_election.cc:288] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c6534fa12d84b19ab50ca02489beb17 (127.0.98.1:42197), eb6f7eb3e75b4d74abb614758f672e61 (127.0.98.2:42313)
I20260619 12:29:59.935802 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.931052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39345 (local address 127.0.98.2:42313)
0619 12:29:59.931173 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:59.931179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:59.931198 (+ 19us) server_negotiation.cc:407] Connection header received
0619 12:29:59.931427 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:59.931432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:59.931503 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:59.931646 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:29:59.931989 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.932751 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.934369 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.934999 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.935150 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.935223 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:59.935557 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 12:29:59.935669 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 12:29:59.935802 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.931052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51173 (local address 127.0.98.1:42197)
0619 12:29:59.931174 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 12:29:59.931179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:29:59.931442 (+ 263us) server_negotiation.cc:407] Connection header received
0619 12:29:59.931568 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:29:59.931573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:29:59.931639 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:29:59.931720 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:29:59.932371 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.933149 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.934520 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.935188 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.935296 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.935351 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 12:29:59.935557 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 12:29:59.935658 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 12:29:59.935822 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.930944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42197 (local address 127.0.98.3:51173)
0619 12:29:59.931323 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:59.931338 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:59.931434 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:59.931732 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:59.931734 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:59.931745 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:29:59.932249 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.932256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.933250 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.933253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.934366 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.934400 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.935293 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.935296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.935382 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.935422 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 12:29:59.935490 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 12:29:59.935620 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":23,"thread_start_us":100,"threads_started":1}
I20260619 12:29:59.935822 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:29:59.930909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42313 (local address 127.0.98.3:39345)
0619 12:29:59.931120 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 12:29:59.931135 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:29:59.931235 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:29:59.931662 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:29:59.931664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:29:59.931677 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:29:59.931844 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.931853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.932863 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.932866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.934215 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:29:59.934225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:29:59.935195 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:29:59.935199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:29:59.935340 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:29:59.935371 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 12:29:59.935465 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 12:29:59.935620 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":65}
I20260619 12:29:59.936523 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9eea2338fe480596de01b205e616fe" candidate_uuid: "9db27fd360884e7e92427fe6a322c61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c6534fa12d84b19ab50ca02489beb17" is_pre_election: true
I20260619 12:29:59.936712 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9eea2338fe480596de01b205e616fe" candidate_uuid: "9db27fd360884e7e92427fe6a322c61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6f7eb3e75b4d74abb614758f672e61" is_pre_election: true
I20260619 12:29:59.936863 2298 raft_consensus.cc:2454] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9db27fd360884e7e92427fe6a322c61f in term 0.
I20260619 12:29:59.937037 2438 raft_consensus.cc:2454] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9db27fd360884e7e92427fe6a322c61f in term 0.
I20260619 12:29:59.937463 2512 leader_election.cc:302] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6534fa12d84b19ab50ca02489beb17, 9db27fd360884e7e92427fe6a322c61f; no voters:
I20260619 12:29:59.937752 2653 raft_consensus.cc:2781] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:29:59.937844 2653 raft_consensus.cc:482] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:29:59.937884 2653 raft_consensus.cc:3037] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:29:59.997887 2653 raft_consensus.cc:504] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:29:59.998570 2653 leader_election.cc:288] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [CANDIDATE]: Term 1 election: Requested vote from peers 3c6534fa12d84b19ab50ca02489beb17 (127.0.98.1:42197), eb6f7eb3e75b4d74abb614758f672e61 (127.0.98.2:42313)
I20260619 12:29:59.999279 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9eea2338fe480596de01b205e616fe" candidate_uuid: "9db27fd360884e7e92427fe6a322c61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c6534fa12d84b19ab50ca02489beb17"
I20260619 12:29:59.999248 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9eea2338fe480596de01b205e616fe" candidate_uuid: "9db27fd360884e7e92427fe6a322c61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6f7eb3e75b4d74abb614758f672e61"
I20260619 12:29:59.999431 2298 raft_consensus.cc:3037] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:29:59.999431 2438 raft_consensus.cc:3037] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:00.000869 2438 raft_consensus.cc:2454] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9db27fd360884e7e92427fe6a322c61f in term 1.
I20260619 12:30:00.000870 2298 raft_consensus.cc:2454] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9db27fd360884e7e92427fe6a322c61f in term 1.
I20260619 12:30:00.001405 2512 leader_election.cc:302] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6534fa12d84b19ab50ca02489beb17, 9db27fd360884e7e92427fe6a322c61f; no voters:
I20260619 12:30:00.001645 2653 raft_consensus.cc:2781] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:00.001992 2653 raft_consensus.cc:686] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [term 1 LEADER]: Becoming Leader. State: Replica: 9db27fd360884e7e92427fe6a322c61f, State: Running, Role: LEADER
I20260619 12:30:00.002408 2653 consensus_queue.cc:237] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.006017 2141 catalog_manager.cc:5495] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9db27fd360884e7e92427fe6a322c61f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9db27fd360884e7e92427fe6a322c61f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } health_report { overall_health: HEALTHY } } }
Jun 19 12:30:00 dist-test-slave-fg3h krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:00.048071 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.038898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47670 (local address 127.0.98.62:43019)
0619 12:30:00.039037 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:00.039042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:00.039234 (+ 192us) server_negotiation.cc:407] Connection header received
0619 12:30:00.039356 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:00.039358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:00.039398 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:00.039468 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:00.040207 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.040715 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.041679 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.041888 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.042009 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.044488 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:00.044491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:00.044494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:00.044514 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:00.046202 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:00.046890 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:00.046893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:00.046894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:00.046982 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:00.047236 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:00.047238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:00.047239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:00.047413 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:00.047492 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:00.047792 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 12:30:00.047929 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":60}
I20260619 12:30:00.048072 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.038833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.0.1:47670)
0619 12:30:00.039134 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:00.039149 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:00.039263 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:00.039523 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:00.039525 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:00.039986 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:00.040086 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:00.040092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.040854 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.040856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:00.041561 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:00.041571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.042048 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.042053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:00.042167 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.042856 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:00.042869 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:00.044350 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:00.046433 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:00.046444 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:00.046451 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:00.046772 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:00.047069 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:00.047071 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:00.047073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:00.047155 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:00.047507 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:00.047513 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:00.047623 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 12:30:00.047732 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 12:30:00.047872 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":125,"threads_started":1}
I20260619 12:30:00.052213 2141 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47670:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 12:30:00.052894 2141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:30:00.057984 2278 tablet_service.cc:1467] Processing CreateTablet for tablet ed32c4646be24253b8acfb86ad305190 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=437c8e494db54d7da05894a98f113ea5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:30:00.058054 2418 tablet_service.cc:1467] Processing CreateTablet for tablet ed32c4646be24253b8acfb86ad305190 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=437c8e494db54d7da05894a98f113ea5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:30:00.058015 2558 tablet_service.cc:1467] Processing CreateTablet for tablet ed32c4646be24253b8acfb86ad305190 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=437c8e494db54d7da05894a98f113ea5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:30:00.058709 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed32c4646be24253b8acfb86ad305190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:00.058712 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed32c4646be24253b8acfb86ad305190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:00.058712 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed32c4646be24253b8acfb86ad305190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:00.061199 2647 tablet_bootstrap.cc:492] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f: Bootstrap starting.
I20260619 12:30:00.061561 2648 tablet_bootstrap.cc:492] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17: Bootstrap starting.
I20260619 12:30:00.062240 2647 tablet_bootstrap.cc:654] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:00.062544 2648 tablet_bootstrap.cc:654] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:00.063261 2649 tablet_bootstrap.cc:492] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61: Bootstrap starting.
I20260619 12:30:00.064042 2647 tablet_bootstrap.cc:492] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f: No bootstrap required, opened a new log
I20260619 12:30:00.064119 2647 ts_tablet_manager.cc:1397] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 12:30:00.064211 2649 tablet_bootstrap.cc:654] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:00.064621 2647 raft_consensus.cc:348] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.064749 2647 raft_consensus.cc:374] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:00.064783 2647 raft_consensus.cc:729] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db27fd360884e7e92427fe6a322c61f, State: Initialized, Role: FOLLOWER
I20260619 12:30:00.064908 2647 consensus_queue.cc:260] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.065197 2647 ts_tablet_manager.cc:1428] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 12:30:00.065214 2653 tablet_replica.cc:439] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } } }
I20260619 12:30:00.065335 2653 tablet_replica.cc:442] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:30:00.065508 2649 tablet_bootstrap.cc:492] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61: No bootstrap required, opened a new log
I20260619 12:30:00.065578 2649 ts_tablet_manager.cc:1397] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:30:00.066212 2648 tablet_bootstrap.cc:492] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17: No bootstrap required, opened a new log
I20260619 12:30:00.066217 2649 raft_consensus.cc:348] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.066313 2648 ts_tablet_manager.cc:1397] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260619 12:30:00.066354 2649 raft_consensus.cc:374] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:00.066398 2649 raft_consensus.cc:729] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6f7eb3e75b4d74abb614758f672e61, State: Initialized, Role: FOLLOWER
I20260619 12:30:00.066534 2649 consensus_queue.cc:260] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.066761 2648 raft_consensus.cc:348] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.066802 2654 tablet_replica.cc:439] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } } }
I20260619 12:30:00.066891 2648 raft_consensus.cc:374] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:00.066880 2649 ts_tablet_manager.cc:1428] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 12:30:00.066927 2648 raft_consensus.cc:729] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c6534fa12d84b19ab50ca02489beb17, State: Initialized, Role: FOLLOWER
I20260619 12:30:00.066951 2654 tablet_replica.cc:442] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:30:00.067036 2648 consensus_queue.cc:260] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.067991 2655 tablet_replica.cc:439] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } } }
I20260619 12:30:00.068135 2655 tablet_replica.cc:442] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:30:00.068405 2648 ts_tablet_manager.cc:1428] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 12:30:00.321349 2653 raft_consensus.cc:482] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:30:00.321518 2653 raft_consensus.cc:504] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.322002 2653 leader_election.cc:288] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c6534fa12d84b19ab50ca02489beb17 (127.0.98.1:42197), eb6f7eb3e75b4d74abb614758f672e61 (127.0.98.2:42313)
I20260619 12:30:00.322501 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32c4646be24253b8acfb86ad305190" candidate_uuid: "9db27fd360884e7e92427fe6a322c61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c6534fa12d84b19ab50ca02489beb17" is_pre_election: true
I20260619 12:30:00.322535 2654 raft_consensus.cc:482] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:30:00.322582 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32c4646be24253b8acfb86ad305190" candidate_uuid: "9db27fd360884e7e92427fe6a322c61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6f7eb3e75b4d74abb614758f672e61" is_pre_election: true
I20260619 12:30:00.322661 2298 raft_consensus.cc:2454] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9db27fd360884e7e92427fe6a322c61f in term 0.
I20260619 12:30:00.322671 2654 raft_consensus.cc:504] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.323043 2512 leader_election.cc:302] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6534fa12d84b19ab50ca02489beb17, 9db27fd360884e7e92427fe6a322c61f; no voters:
I20260619 12:30:00.323122 2438 raft_consensus.cc:2454] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9db27fd360884e7e92427fe6a322c61f in term 0.
I20260619 12:30:00.323220 2653 raft_consensus.cc:2781] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:30:00.323274 2653 raft_consensus.cc:482] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:30:00.323321 2653 raft_consensus.cc:3037] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:00.324041 2654 leader_election.cc:288] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c6534fa12d84b19ab50ca02489beb17 (127.0.98.1:42197), 9db27fd360884e7e92427fe6a322c61f (127.0.98.3:36531)
I20260619 12:30:00.324651 2653 raft_consensus.cc:504] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.325022 2653 leader_election.cc:288] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f [CANDIDATE]: Term 1 election: Requested vote from peers 3c6534fa12d84b19ab50ca02489beb17 (127.0.98.1:42197), eb6f7eb3e75b4d74abb614758f672e61 (127.0.98.2:42313)
I20260619 12:30:00.325340 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32c4646be24253b8acfb86ad305190" candidate_uuid: "9db27fd360884e7e92427fe6a322c61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6f7eb3e75b4d74abb614758f672e61"
I20260619 12:30:00.325455 2438 raft_consensus.cc:3037] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:00.325500 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32c4646be24253b8acfb86ad305190" candidate_uuid: "9db27fd360884e7e92427fe6a322c61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c6534fa12d84b19ab50ca02489beb17"
I20260619 12:30:00.325634 2298 raft_consensus.cc:3037] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:00.326575 2298 raft_consensus.cc:2454] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9db27fd360884e7e92427fe6a322c61f in term 1.
I20260619 12:30:00.326912 2512 leader_election.cc:302] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6534fa12d84b19ab50ca02489beb17, 9db27fd360884e7e92427fe6a322c61f; no voters:
I20260619 12:30:00.327055 2438 raft_consensus.cc:2454] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9db27fd360884e7e92427fe6a322c61f in term 1.
I20260619 12:30:00.327090 2653 raft_consensus.cc:2781] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:00.327168 2653 raft_consensus.cc:686] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f [term 1 LEADER]: Becoming Leader. State: Replica: 9db27fd360884e7e92427fe6a322c61f, State: Running, Role: LEADER
I20260619 12:30:00.327333 2653 consensus_queue.cc:237] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.328049 2657 tablet_replica.cc:439] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f: TxnStatusTablet state changed. Reason: New leader 9db27fd360884e7e92427fe6a322c61f. Latest consensus state: current_term: 1 leader_uuid: "9db27fd360884e7e92427fe6a322c61f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } } }
I20260619 12:30:00.328162 2657 tablet_replica.cc:442] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:30:00.328785 2672 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 12:30:00.328953 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.324309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55483 (local address 127.0.98.1:42197)
0619 12:30:00.324472 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:00.324478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:00.324557 (+ 79us) server_negotiation.cc:407] Connection header received
0619 12:30:00.324711 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:00.324715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:00.324771 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:00.324844 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:00.325288 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.326043 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.327659 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.328195 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.328275 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.328336 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:00.328729 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 12:30:00.328819 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 12:30:00.329259 2672 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 12:30:00.329326 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.324230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42197 (local address 127.0.98.2:55483)
0619 12:30:00.324469 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:00.324486 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:00.324604 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:00.324975 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:00.324980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:00.325002 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:00.325159 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:00.325168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.326200 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.326203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:00.327523 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:00.327532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.328468 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.328475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:00.328576 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.328600 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 12:30:00.329101 (+ 501us) client_negotiation.cc:240] Negotiation successful
0619 12:30:00.329181 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":81,"threads_started":1}
I20260619 12:30:00.329433 2141 catalog_manager.cc:5495] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9db27fd360884e7e92427fe6a322c61f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9db27fd360884e7e92427fe6a322c61f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } health_report { overall_health: HEALTHY } } }
I20260619 12:30:00.330407 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.324576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36531 (local address 127.0.98.2:55395)
0619 12:30:00.324894 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:00.324908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:00.324994 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:00.326988 (+ 1994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:00.326991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:00.327009 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:00.327132 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:00.327141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.328081 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.328085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:00.329162 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:00.329173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.330010 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.330014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:00.330108 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.330130 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:30:00.330210 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 12:30:00.330271 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":57,"threads_started":1}
I20260619 12:30:00.330443 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.324845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55395 (local address 127.0.98.3:36531)
0619 12:30:00.325275 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:00.325289 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:00.325301 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:30:00.325339 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:00.325353 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:00.326791 (+ 1438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:00.326875 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:00.327265 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.327951 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.329295 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.329887 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.329993 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.330050 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:00.330241 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 12:30:00.330315 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":71,"threads_started":1}
I20260619 12:30:00.331442 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32c4646be24253b8acfb86ad305190" candidate_uuid: "eb6f7eb3e75b4d74abb614758f672e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db27fd360884e7e92427fe6a322c61f" is_pre_election: true
I20260619 12:30:00.332209 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed32c4646be24253b8acfb86ad305190" candidate_uuid: "eb6f7eb3e75b4d74abb614758f672e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c6534fa12d84b19ab50ca02489beb17" is_pre_election: true
I20260619 12:30:00.332401 2298 raft_consensus.cc:2379] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb6f7eb3e75b4d74abb614758f672e61 in current term 1: Already voted for candidate 9db27fd360884e7e92427fe6a322c61f in this term.
I20260619 12:30:00.332808 2372 leader_election.cc:302] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6f7eb3e75b4d74abb614758f672e61; no voters: 3c6534fa12d84b19ab50ca02489beb17, 9db27fd360884e7e92427fe6a322c61f
I20260619 12:30:00.333040 2654 raft_consensus.cc:2726] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 12:30:00.367466 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.363132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59110 (local address 127.0.98.3:36531)
0619 12:30:00.363277 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:00.363282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:00.363299 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:30:00.363412 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:00.363415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:00.363477 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:00.363618 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:30:00.364017 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.364790 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.365731 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.366110 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.366223 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.366514 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:30:00.367060 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:30:00.367170 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:00.367272 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 12:30:00.367328 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 12:30:00.367516 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.362959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36531 (local address 127.0.0.1:59110)
0619 12:30:00.363129 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:00.363146 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:00.363264 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:00.363706 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:00.363710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:00.363766 (+ 56us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:30:00.363892 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:00.363899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.364898 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.364901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:00.365626 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:00.365635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.366249 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.366252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:00.366323 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.366370 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:30:00.367162 (+ 792us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:30:00.367176 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 12:30:00.367240 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 12:30:00.367316 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 12:30:00.372694 2298 raft_consensus.cc:1261] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17 [term 1 FOLLOWER]: Refusing update from remote peer 9db27fd360884e7e92427fe6a322c61f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:00.372735 2438 raft_consensus.cc:1261] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61 [term 1 FOLLOWER]: Refusing update from remote peer 9db27fd360884e7e92427fe6a322c61f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:00.373380 2657 consensus_queue.cc:1035] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:00.373570 2653 consensus_queue.cc:1035] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:00.376822 2654 tablet_replica.cc:439] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61: TxnStatusTablet state changed. Reason: New leader 9db27fd360884e7e92427fe6a322c61f. Latest consensus state: current_term: 1 leader_uuid: "9db27fd360884e7e92427fe6a322c61f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } } }
I20260619 12:30:00.376951 2654 tablet_replica.cc:442] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:30:00.376878 2655 tablet_replica.cc:439] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17: TxnStatusTablet state changed. Reason: New leader 9db27fd360884e7e92427fe6a322c61f. Latest consensus state: current_term: 1 leader_uuid: "9db27fd360884e7e92427fe6a322c61f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } } }
I20260619 12:30:00.376994 2655 tablet_replica.cc:442] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:30:00.380101 2675 mvcc.cc:205] Tried to move back new op lower bound from 7298548532719448064 to 7298548532541898752. Current Snapshot: MvccSnapshot[applied={T|T < 7298548532719448064}]
I20260619 12:30:00.380357 2653 tablet_replica.cc:439] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9db27fd360884e7e92427fe6a322c61f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } } }
I20260619 12:30:00.380477 2653 tablet_replica.cc:442] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:30:00.381340 2657 tablet_replica.cc:439] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9db27fd360884e7e92427fe6a322c61f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } } }
I20260619 12:30:00.381464 2657 tablet_replica.cc:442] T ed32c4646be24253b8acfb86ad305190 P 9db27fd360884e7e92427fe6a322c61f: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:30:00.382479 2654 tablet_replica.cc:439] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9db27fd360884e7e92427fe6a322c61f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } } }
I20260619 12:30:00.382603 2654 tablet_replica.cc:442] T ed32c4646be24253b8acfb86ad305190 P eb6f7eb3e75b4d74abb614758f672e61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:30:00.385075 2677 mvcc.cc:205] Tried to move back new op lower bound from 7298548532719448064 to 7298548532541898752. Current Snapshot: MvccSnapshot[applied={T|T < 7298548532719448064}]
I20260619 12:30:00.385793 2655 tablet_replica.cc:439] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9db27fd360884e7e92427fe6a322c61f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } } }
I20260619 12:30:00.385919 2655 tablet_replica.cc:442] T ed32c4646be24253b8acfb86ad305190 P 3c6534fa12d84b19ab50ca02489beb17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:30:00.393426 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.390036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59120 (local address 127.0.98.3:36531)
0619 12:30:00.390142 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:00.390147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:00.390319 (+ 172us) server_negotiation.cc:407] Connection header received
0619 12:30:00.390406 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:00.390409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:00.390454 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:00.390532 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:30:00.390738 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.391508 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.392318 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.392654 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.392755 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.392952 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:30:00.393026 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:30:00.393116 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:00.393250 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 12:30:00.393298 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 12:30:00.404469 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.400405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42823 (local address 127.0.98.3:36531)
0619 12:30:00.400524 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:00.400528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:00.400688 (+ 160us) server_negotiation.cc:407] Connection header received
0619 12:30:00.400798 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:00.400800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:00.400838 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:00.400891 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:30:00.401178 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.401647 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.403384 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.403589 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.403670 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.404040 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:30:00.404104 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:30:00.404172 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:00.404283 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 12:30:00.404358 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":50}
I20260619 12:30:00.404521 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.400350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36531 (local address 127.0.98.3:42823)
0619 12:30:00.400602 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:00.400617 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:00.400716 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:00.400922 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:00.400924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:00.400942 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:30:00.401046 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:00.401051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.401753 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.401756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:00.403260 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:00.403271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.403701 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.403705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:00.403799 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.403847 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:30:00.404198 (+ 351us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:30:00.404204 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 12:30:00.404269 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 12:30:00.404367 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":3,"thread_start_us":107,"threads_started":1}
I20260619 12:30:00.411092 2438 raft_consensus.cc:1261] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61 [term 1 FOLLOWER]: Refusing update from remote peer 9db27fd360884e7e92427fe6a322c61f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:00.411276 2298 raft_consensus.cc:1261] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17 [term 1 FOLLOWER]: Refusing update from remote peer 9db27fd360884e7e92427fe6a322c61f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:00.411470 2653 consensus_queue.cc:1035] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:00.411669 2657 consensus_queue.cc:1035] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:00.449851 2140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:00.450065 2140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:00.452292 2140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe: Sending DeleteTablet for 3 replicas of tablet 6e9eea2338fe480596de01b205e616fe
I20260619 12:30:00.452968 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e9eea2338fe480596de01b205e616fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43324
I20260619 12:30:00.453038 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e9eea2338fe480596de01b205e616fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52692
I20260619 12:30:00.453521 2693 tablet_replica.cc:330] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61: stopping tablet replica
I20260619 12:30:00.453648 2140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 12:30:00.453890 2693 raft_consensus.cc:2229] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 12:30:00.454072 2140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:30:00.454095 2693 raft_consensus.cc:2258] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:00.454275 2692 tablet_replica.cc:330] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17: stopping tablet replica
I20260619 12:30:00.454581 2692 raft_consensus.cc:2229] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:00.454780 2692 raft_consensus.cc:2258] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:00.454839 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e9eea2338fe480596de01b205e616fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59096
I20260619 12:30:00.455950 2694 tablet_replica.cc:330] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f: stopping tablet replica
I20260619 12:30:00.456053 2692 ts_tablet_manager.cc:1905] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:00.456208 2694 raft_consensus.cc:2229] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:30:00.456508 2694 raft_consensus.cc:2258] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:00.457978 2694 ts_tablet_manager.cc:1905] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:00.458865 2693 ts_tablet_manager.cc:1905] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:00.459604 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 19aaa2ae92de470ba358b5f675e655ed (DEFAULT_TABLE table=test-table [id=96cc5c2694ab466db2712078427eac80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:00.459614 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 19aaa2ae92de470ba358b5f675e655ed (DEFAULT_TABLE table=test-table [id=96cc5c2694ab466db2712078427eac80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:00.459852 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19aaa2ae92de470ba358b5f675e655ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:00.459880 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19aaa2ae92de470ba358b5f675e655ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:00.460917 2694 ts_tablet_manager.cc:1918] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:30:00.460999 2694 log.cc:1196] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/6e9eea2338fe480596de01b205e616fe
I20260619 12:30:00.461709 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 19aaa2ae92de470ba358b5f675e655ed (DEFAULT_TABLE table=test-table [id=96cc5c2694ab466db2712078427eac80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:00.461951 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19aaa2ae92de470ba358b5f675e655ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:00.462296 2647 tablet_bootstrap.cc:492] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f: Bootstrap starting.
I20260619 12:30:00.463249 2647 tablet_bootstrap.cc:654] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:00.463327 2692 ts_tablet_manager.cc:1918] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:30:00.463416 2692 log.cc:1196] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/wal/wals/6e9eea2338fe480596de01b205e616fe
I20260619 12:30:00.463737 2692 ts_tablet_manager.cc:1939] T 6e9eea2338fe480596de01b205e616fe P 3c6534fa12d84b19ab50ca02489beb17: Deleting consensus metadata
I20260619 12:30:00.464135 2648 tablet_bootstrap.cc:492] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17: Bootstrap starting.
I20260619 12:30:00.464421 2647 tablet_bootstrap.cc:492] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f: No bootstrap required, opened a new log
I20260619 12:30:00.464501 2647 ts_tablet_manager.cc:1397] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:30:00.464951 2128 catalog_manager.cc:4841] TS 3c6534fa12d84b19ab50ca02489beb17 (127.0.98.1:42197): tablet 6e9eea2338fe480596de01b205e616fe (table test-table [id=bde3f81170c94919b3a9672027a072bd]) successfully deleted
I20260619 12:30:00.464960 2647 raft_consensus.cc:348] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.465067 2648 tablet_bootstrap.cc:654] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:00.465111 2647 raft_consensus.cc:374] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:00.465145 2647 raft_consensus.cc:729] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db27fd360884e7e92427fe6a322c61f, State: Initialized, Role: FOLLOWER
I20260619 12:30:00.465312 2694 ts_tablet_manager.cc:1939] T 6e9eea2338fe480596de01b205e616fe P 9db27fd360884e7e92427fe6a322c61f: Deleting consensus metadata
I20260619 12:30:00.465276 2647 consensus_queue.cc:260] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.465576 2647 ts_tablet_manager.cc:1428] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 12:30:00.465723 2649 tablet_bootstrap.cc:492] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61: Bootstrap starting.
I20260619 12:30:00.466600 2649 tablet_bootstrap.cc:654] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:00.467173 2128 catalog_manager.cc:4841] TS 9db27fd360884e7e92427fe6a322c61f (127.0.98.3:36531): tablet 6e9eea2338fe480596de01b205e616fe (table test-table [id=bde3f81170c94919b3a9672027a072bd]) successfully deleted
I20260619 12:30:00.467559 2648 tablet_bootstrap.cc:492] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17: No bootstrap required, opened a new log
I20260619 12:30:00.467653 2648 ts_tablet_manager.cc:1397] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260619 12:30:00.468077 2648 raft_consensus.cc:348] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.468189 2648 raft_consensus.cc:374] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:00.468230 2648 raft_consensus.cc:729] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c6534fa12d84b19ab50ca02489beb17, State: Initialized, Role: FOLLOWER
I20260619 12:30:00.468360 2648 consensus_queue.cc:260] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.468622 2648 ts_tablet_manager.cc:1428] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 12:30:00.469149 2649 tablet_bootstrap.cc:492] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61: No bootstrap required, opened a new log
I20260619 12:30:00.469244 2649 ts_tablet_manager.cc:1397] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 12:30:00.469265 2693 ts_tablet_manager.cc:1918] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:30:00.469369 2693 log.cc:1196] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/6e9eea2338fe480596de01b205e616fe
I20260619 12:30:00.469676 2649 raft_consensus.cc:348] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.469800 2649 raft_consensus.cc:374] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:00.469832 2649 raft_consensus.cc:729] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6f7eb3e75b4d74abb614758f672e61, State: Initialized, Role: FOLLOWER
I20260619 12:30:00.469955 2649 consensus_queue.cc:260] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.470194 2649 ts_tablet_manager.cc:1428] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:30:00.470422 2693 ts_tablet_manager.cc:1939] T 6e9eea2338fe480596de01b205e616fe P eb6f7eb3e75b4d74abb614758f672e61: Deleting consensus metadata
I20260619 12:30:00.471562 2125 catalog_manager.cc:4841] TS eb6f7eb3e75b4d74abb614758f672e61 (127.0.98.2:42313): tablet 6e9eea2338fe480596de01b205e616fe (table test-table [id=bde3f81170c94919b3a9672027a072bd]) successfully deleted
I20260619 12:30:00.538743 2655 raft_consensus.cc:482] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:30:00.538880 2655 raft_consensus.cc:504] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.540078 2655 leader_election.cc:288] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6f7eb3e75b4d74abb614758f672e61 (127.0.98.2:42313), 9db27fd360884e7e92427fe6a322c61f (127.0.98.3:36531)
I20260619 12:30:00.544468 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.540327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36531 (local address 127.0.98.1:53887)
0619 12:30:00.540636 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:00.540653 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:00.540778 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:00.541131 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:00.541134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:00.541154 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:00.541271 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:00.541277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.542198 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.542201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:00.543315 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:00.543328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.544035 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.544039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:00.544143 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.544169 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 12:30:00.544243 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 12:30:00.544310 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"spinlock_wait_cycles":10752,"thread_start_us":107,"threads_started":1}
I20260619 12:30:00.544471 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.540676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53887 (local address 127.0.98.3:36531)
0619 12:30:00.540784 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:00.540788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:00.540804 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:30:00.540862 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:00.540865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:00.540911 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:00.540986 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:00.541449 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.542067 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.543459 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.543881 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.543998 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.544063 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:00.544276 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 12:30:00.544348 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 12:30:00.544901 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19aaa2ae92de470ba358b5f675e655ed" candidate_uuid: "3c6534fa12d84b19ab50ca02489beb17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db27fd360884e7e92427fe6a322c61f" is_pre_election: true
I20260619 12:30:00.545020 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.540235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42313 (local address 127.0.98.1:55627)
0619 12:30:00.540636 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:00.540653 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:00.540778 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:00.541266 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:00.541269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:00.541287 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:00.541448 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:00.541455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.542556 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.542559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:00.543750 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:00.543761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.544617 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.544621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:00.544722 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.544744 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:30:00.544797 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 12:30:00.544881 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"spinlock_wait_cycles":19200,"thread_start_us":123,"threads_started":1}
I20260619 12:30:00.545148 2571 raft_consensus.cc:2454] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c6534fa12d84b19ab50ca02489beb17 in term 0.
I20260619 12:30:00.545506 2232 leader_election.cc:302] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6534fa12d84b19ab50ca02489beb17, 9db27fd360884e7e92427fe6a322c61f; no voters:
I20260619 12:30:00.545651 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.540365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55627 (local address 127.0.98.2:42313)
0619 12:30:00.541002 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:00.541009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:00.541021 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:30:00.541062 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:00.541064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:00.541159 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:00.541254 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:00.541580 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.542405 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.543886 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.544474 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.545308 (+ 834us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.545403 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:00.545451 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 12:30:00.545518 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":119,"threads_started":1}
I20260619 12:30:00.545768 2655 raft_consensus.cc:2781] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:30:00.545823 2655 raft_consensus.cc:482] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:30:00.545851 2655 raft_consensus.cc:3037] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:00.546073 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19aaa2ae92de470ba358b5f675e655ed" candidate_uuid: "3c6534fa12d84b19ab50ca02489beb17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6f7eb3e75b4d74abb614758f672e61" is_pre_election: true
I20260619 12:30:00.546203 2437 raft_consensus.cc:2454] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c6534fa12d84b19ab50ca02489beb17 in term 0.
I20260619 12:30:00.547549 2655 raft_consensus.cc:504] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.547899 2655 leader_election.cc:288] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [CANDIDATE]: Term 1 election: Requested vote from peers eb6f7eb3e75b4d74abb614758f672e61 (127.0.98.2:42313), 9db27fd360884e7e92427fe6a322c61f (127.0.98.3:36531)
I20260619 12:30:00.548203 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19aaa2ae92de470ba358b5f675e655ed" candidate_uuid: "3c6534fa12d84b19ab50ca02489beb17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6f7eb3e75b4d74abb614758f672e61"
I20260619 12:30:00.548297 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19aaa2ae92de470ba358b5f675e655ed" candidate_uuid: "3c6534fa12d84b19ab50ca02489beb17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db27fd360884e7e92427fe6a322c61f"
I20260619 12:30:00.548336 2437 raft_consensus.cc:3037] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:00.548404 2571 raft_consensus.cc:3037] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:00.549510 2571 raft_consensus.cc:2454] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c6534fa12d84b19ab50ca02489beb17 in term 1.
I20260619 12:30:00.549791 2232 leader_election.cc:302] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6534fa12d84b19ab50ca02489beb17, 9db27fd360884e7e92427fe6a322c61f; no voters:
I20260619 12:30:00.549896 2437 raft_consensus.cc:2454] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c6534fa12d84b19ab50ca02489beb17 in term 1.
I20260619 12:30:00.549942 2655 raft_consensus.cc:2781] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:00.550108 2655 raft_consensus.cc:686] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [term 1 LEADER]: Becoming Leader. State: Replica: 3c6534fa12d84b19ab50ca02489beb17, State: Running, Role: LEADER
I20260619 12:30:00.550333 2655 consensus_queue.cc:237] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:00.552644 2141 catalog_manager.cc:5495] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c6534fa12d84b19ab50ca02489beb17 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3c6534fa12d84b19ab50ca02489beb17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } health_report { overall_health: UNKNOWN } } }
I20260619 12:30:00.575657 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.571548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.0.98.1:42197)
0619 12:30:00.571668 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:00.571671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:00.571683 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:30:00.571824 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:00.571827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:00.571877 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:00.571968 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:30:00.572204 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.572705 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.573932 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.574307 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.574403 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.574646 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:30:00.575213 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:30:00.575324 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:00.575455 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 12:30:00.575521 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 12:30:00.586519 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.582906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36531 (local address 127.0.98.1:47997)
0619 12:30:00.583221 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:00.583234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:00.583318 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:00.583551 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:00.583553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:00.583559 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:30:00.583651 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:00.583655 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.584606 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.584608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:00.585273 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:00.585281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.585834 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.585837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:00.585949 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.585996 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:30:00.586279 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:30:00.586284 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:30:00.586333 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 12:30:00.586393 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":85,"threads_started":1}
I20260619 12:30:00.586534 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.582968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47997 (local address 127.0.98.3:36531)
0619 12:30:00.583087 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:00.583091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:00.583301 (+ 210us) server_negotiation.cc:407] Connection header received
0619 12:30:00.583403 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:00.583406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:00.583455 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:00.583536 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:30:00.583742 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.584498 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.585381 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.585719 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.585823 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.586124 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:30:00.586199 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:30:00.586269 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:00.586356 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 12:30:00.586406 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:30:00.592715 2437 raft_consensus.cc:1261] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61 [term 1 FOLLOWER]: Refusing update from remote peer 3c6534fa12d84b19ab50ca02489beb17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:00.592869 2571 raft_consensus.cc:1261] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f [term 1 FOLLOWER]: Refusing update from remote peer 3c6534fa12d84b19ab50ca02489beb17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:00.593200 2655 consensus_queue.cc:1035] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:00.593374 2701 consensus_queue.cc:1035] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:00.597633 2676 mvcc.cc:205] Tried to move back new op lower bound from 7298548533623152640 to 7298548533456261120. Current Snapshot: MvccSnapshot[applied={T|T < 7298548533623152640}]
I20260619 12:30:00.611685 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.608423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42197 (local address 127.0.98.3:36459)
0619 12:30:00.608538 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:00.608553 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:00.608633 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:00.608913 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:00.608915 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:00.608922 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:30:00.609017 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:00.609023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.609766 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.609772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:00.610379 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:00.610388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.610984 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.610987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:00.611053 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.611077 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:30:00.611379 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:30:00.611385 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 12:30:00.611478 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 12:30:00.611563 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 12:30:00.611685 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:00.608562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36459 (local address 127.0.98.1:42197)
0619 12:30:00.608710 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:00.608717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:00.608730 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:30:00.608766 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:00.608769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:00.608814 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:00.608892 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:30:00.609170 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.609661 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.610519 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:00.610787 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:00.610897 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:00.611195 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:30:00.611277 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:30:00.611343 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:00.611495 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 12:30:00.611549 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 12:30:00.612571 2677 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 12:30:02.175586 2140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:02.175786 2140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:02.178189 2140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe: Sending DeleteTablet for 3 replicas of tablet 19aaa2ae92de470ba358b5f675e655ed
I20260619 12:30:02.178727 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet 19aaa2ae92de470ba358b5f675e655ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43324
I20260619 12:30:02.178833 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet 19aaa2ae92de470ba358b5f675e655ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59096
I20260619 12:30:02.178879 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet 19aaa2ae92de470ba358b5f675e655ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52692
I20260619 12:30:02.179358 2732 tablet_replica.cc:330] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17: stopping tablet replica
I20260619 12:30:02.180003 2732 raft_consensus.cc:2229] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:30:02.180344 2732 raft_consensus.cc:2258] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:02.181144 2732 ts_tablet_manager.cc:1905] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:02.181488 2734 tablet_replica.cc:330] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61: stopping tablet replica
I20260619 12:30:02.181705 2734 raft_consensus.cc:2229] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:02.181893 2734 raft_consensus.cc:2258] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:02.182082 2733 tablet_replica.cc:330] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f: stopping tablet replica
I20260619 12:30:02.182250 2733 raft_consensus.cc:2229] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:02.182413 2733 raft_consensus.cc:2258] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:02.182605 2734 ts_tablet_manager.cc:1905] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:02.183079 2733 ts_tablet_manager.cc:1905] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:02.183945 2732 ts_tablet_manager.cc:1918] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 12:30:02.184041 2732 log.cc:1196] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/wal/wals/19aaa2ae92de470ba358b5f675e655ed
I20260619 12:30:02.184314 2732 ts_tablet_manager.cc:1939] T 19aaa2ae92de470ba358b5f675e655ed P 3c6534fa12d84b19ab50ca02489beb17: Deleting consensus metadata
I20260619 12:30:02.185015 2128 catalog_manager.cc:4841] TS 3c6534fa12d84b19ab50ca02489beb17 (127.0.98.1:42197): tablet 19aaa2ae92de470ba358b5f675e655ed (table test-table [id=96cc5c2694ab466db2712078427eac80]) successfully deleted
I20260619 12:30:02.185137 2734 ts_tablet_manager.cc:1918] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 12:30:02.185201 2734 log.cc:1196] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/19aaa2ae92de470ba358b5f675e655ed
I20260619 12:30:02.185467 2734 ts_tablet_manager.cc:1939] T 19aaa2ae92de470ba358b5f675e655ed P eb6f7eb3e75b4d74abb614758f672e61: Deleting consensus metadata
I20260619 12:30:02.185835 2733 ts_tablet_manager.cc:1918] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 12:30:02.185915 2733 log.cc:1196] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/19aaa2ae92de470ba358b5f675e655ed
I20260619 12:30:02.186240 2733 ts_tablet_manager.cc:1939] T 19aaa2ae92de470ba358b5f675e655ed P 9db27fd360884e7e92427fe6a322c61f: Deleting consensus metadata
I20260619 12:30:02.186230 2125 catalog_manager.cc:4841] TS eb6f7eb3e75b4d74abb614758f672e61 (127.0.98.2:42313): tablet 19aaa2ae92de470ba358b5f675e655ed (table test-table [id=96cc5c2694ab466db2712078427eac80]) successfully deleted
I20260619 12:30:02.186851 2128 catalog_manager.cc:4841] TS 9db27fd360884e7e92427fe6a322c61f (127.0.98.3:36531): tablet 19aaa2ae92de470ba358b5f675e655ed (table test-table [id=96cc5c2694ab466db2712078427eac80]) successfully deleted
Jun 19 12:30:02 dist-test-slave-fg3h krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872202, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 12:30:02 dist-test-slave-fg3h krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872202, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:02.212837 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:02.199899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38046 (local address 127.0.98.62:43019)
0619 12:30:02.200210 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:02.200222 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:02.200288 (+ 66us) server_negotiation.cc:407] Connection header received
0619 12:30:02.200431 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:02.200434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:02.200485 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:02.200583 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:02.201344 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:02.201897 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:02.202735 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:02.203141 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:02.203328 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:02.209882 (+ 6554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:02.209885 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:02.209887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:02.209911 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:02.211198 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:02.211611 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:02.211616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:02.211617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:02.211702 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:02.211963 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:02.211968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:02.211969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:02.212152 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:02.212298 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:02.212532 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 12:30:02.212662 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":115,"threads_started":1}
I20260619 12:30:02.215484 2140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:30:02.215932 2140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:30:02.220623 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 4d40a225010049b69e0c0228fd74bec8 (DEFAULT_TABLE table=test-table [id=888c0f79d1304c0a84fc6fe9a1b686be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:02.220688 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 4d40a225010049b69e0c0228fd74bec8 (DEFAULT_TABLE table=test-table [id=888c0f79d1304c0a84fc6fe9a1b686be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:02.220752 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 4d40a225010049b69e0c0228fd74bec8 (DEFAULT_TABLE table=test-table [id=888c0f79d1304c0a84fc6fe9a1b686be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:02.220968 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d40a225010049b69e0c0228fd74bec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:02.220974 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d40a225010049b69e0c0228fd74bec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:02.220999 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d40a225010049b69e0c0228fd74bec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:02.222980 2743 tablet_bootstrap.cc:492] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61: Bootstrap starting.
I20260619 12:30:02.223800 2744 tablet_bootstrap.cc:492] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f: Bootstrap starting.
I20260619 12:30:02.223896 2745 tablet_bootstrap.cc:492] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17: Bootstrap starting.
I20260619 12:30:02.224040 2743 tablet_bootstrap.cc:654] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:02.224737 2744 tablet_bootstrap.cc:654] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:02.224737 2745 tablet_bootstrap.cc:654] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:02.225158 2743 tablet_bootstrap.cc:492] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61: No bootstrap required, opened a new log
I20260619 12:30:02.225262 2743 ts_tablet_manager.cc:1397] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:30:02.225735 2743 raft_consensus.cc:348] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:02.225984 2745 tablet_bootstrap.cc:492] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17: No bootstrap required, opened a new log
I20260619 12:30:02.225971 2743 raft_consensus.cc:374] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:02.226025 2743 raft_consensus.cc:729] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6f7eb3e75b4d74abb614758f672e61, State: Initialized, Role: FOLLOWER
I20260619 12:30:02.226086 2745 ts_tablet_manager.cc:1397] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 12:30:02.226225 2744 tablet_bootstrap.cc:492] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f: No bootstrap required, opened a new log
I20260619 12:30:02.226187 2743 consensus_queue.cc:260] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:02.226310 2744 ts_tablet_manager.cc:1397] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260619 12:30:02.226562 2743 ts_tablet_manager.cc:1428] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:30:02.226548 2745 raft_consensus.cc:348] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:02.226660 2745 raft_consensus.cc:374] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:02.226697 2745 raft_consensus.cc:729] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c6534fa12d84b19ab50ca02489beb17, State: Initialized, Role: FOLLOWER
I20260619 12:30:02.226724 2744 raft_consensus.cc:348] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:02.226828 2744 raft_consensus.cc:374] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:02.226836 2745 consensus_queue.cc:260] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:02.226898 2744 raft_consensus.cc:729] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db27fd360884e7e92427fe6a322c61f, State: Initialized, Role: FOLLOWER
I20260619 12:30:02.227017 2744 consensus_queue.cc:260] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:02.227118 2745 ts_tablet_manager.cc:1428] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 12:30:02.227291 2744 ts_tablet_manager.cc:1428] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 12:30:02.394877 2708 raft_consensus.cc:482] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:30:02.395061 2708 raft_consensus.cc:504] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:02.395495 2708 leader_election.cc:288] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6f7eb3e75b4d74abb614758f672e61 (127.0.98.2:42313), 9db27fd360884e7e92427fe6a322c61f (127.0.98.3:36531)
I20260619 12:30:02.396039 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d40a225010049b69e0c0228fd74bec8" candidate_uuid: "3c6534fa12d84b19ab50ca02489beb17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db27fd360884e7e92427fe6a322c61f" is_pre_election: true
I20260619 12:30:02.396042 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d40a225010049b69e0c0228fd74bec8" candidate_uuid: "3c6534fa12d84b19ab50ca02489beb17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6f7eb3e75b4d74abb614758f672e61" is_pre_election: true
I20260619 12:30:02.396198 2571 raft_consensus.cc:2454] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c6534fa12d84b19ab50ca02489beb17 in term 0.
I20260619 12:30:02.396210 2438 raft_consensus.cc:2454] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c6534fa12d84b19ab50ca02489beb17 in term 0.
I20260619 12:30:02.396545 2229 leader_election.cc:302] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6534fa12d84b19ab50ca02489beb17, eb6f7eb3e75b4d74abb614758f672e61; no voters:
I20260619 12:30:02.396730 2708 raft_consensus.cc:2781] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:30:02.396800 2708 raft_consensus.cc:482] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:30:02.396843 2708 raft_consensus.cc:3037] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:02.397915 2708 raft_consensus.cc:504] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:02.398219 2708 leader_election.cc:288] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [CANDIDATE]: Term 1 election: Requested vote from peers eb6f7eb3e75b4d74abb614758f672e61 (127.0.98.2:42313), 9db27fd360884e7e92427fe6a322c61f (127.0.98.3:36531)
I20260619 12:30:02.398518 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d40a225010049b69e0c0228fd74bec8" candidate_uuid: "3c6534fa12d84b19ab50ca02489beb17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6f7eb3e75b4d74abb614758f672e61"
I20260619 12:30:02.398599 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d40a225010049b69e0c0228fd74bec8" candidate_uuid: "3c6534fa12d84b19ab50ca02489beb17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db27fd360884e7e92427fe6a322c61f"
I20260619 12:30:02.398644 2438 raft_consensus.cc:3037] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:02.398723 2571 raft_consensus.cc:3037] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:02.399786 2438 raft_consensus.cc:2454] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c6534fa12d84b19ab50ca02489beb17 in term 1.
I20260619 12:30:02.399786 2571 raft_consensus.cc:2454] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c6534fa12d84b19ab50ca02489beb17 in term 1.
I20260619 12:30:02.400063 2229 leader_election.cc:302] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6534fa12d84b19ab50ca02489beb17, eb6f7eb3e75b4d74abb614758f672e61; no voters:
I20260619 12:30:02.400249 2708 raft_consensus.cc:2781] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:02.400354 2708 raft_consensus.cc:686] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [term 1 LEADER]: Becoming Leader. State: Replica: 3c6534fa12d84b19ab50ca02489beb17, State: Running, Role: LEADER
I20260619 12:30:02.400506 2708 consensus_queue.cc:237] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:02.402269 2140 catalog_manager.cc:5495] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c6534fa12d84b19ab50ca02489beb17 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3c6534fa12d84b19ab50ca02489beb17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } health_report { overall_health: UNKNOWN } } }
I20260619 12:30:02.424181 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:02.421021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43410 (local address 127.0.98.1:42197)
0619 12:30:02.421296 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:02.421335 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:02.421348 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:30:02.421386 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:02.421388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:02.421430 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:02.421502 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:30:02.421728 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:02.422231 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:02.423085 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:02.423340 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:02.423454 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:02.423710 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:30:02.423793 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:30:02.423871 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:02.423961 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 12:30:02.424014 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":141,"threads_started":1}
I20260619 12:30:02.430248 2571 raft_consensus.cc:1261] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f [term 1 FOLLOWER]: Refusing update from remote peer 3c6534fa12d84b19ab50ca02489beb17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:02.430584 2438 raft_consensus.cc:1261] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61 [term 1 FOLLOWER]: Refusing update from remote peer 3c6534fa12d84b19ab50ca02489beb17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:02.430627 2726 consensus_queue.cc:1035] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:02.430932 2708 consensus_queue.cc:1035] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:02.457553 2140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:02.457724 2140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:02.459537 2140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe: Sending DeleteTablet for 3 replicas of tablet 4d40a225010049b69e0c0228fd74bec8
I20260619 12:30:02.460035 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d40a225010049b69e0c0228fd74bec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43324
I20260619 12:30:02.460151 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d40a225010049b69e0c0228fd74bec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59096
I20260619 12:30:02.460166 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d40a225010049b69e0c0228fd74bec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52692
I20260619 12:30:02.460353 2734 tablet_replica.cc:330] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61: stopping tablet replica
I20260619 12:30:02.460388 2732 tablet_replica.cc:330] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17: stopping tablet replica
I20260619 12:30:02.460458 2734 raft_consensus.cc:2229] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:02.460480 2732 raft_consensus.cc:2229] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:30:02.460558 2734 raft_consensus.cc:2258] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:02.460610 2732 raft_consensus.cc:2258] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:02.460788 2733 tablet_replica.cc:330] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f: stopping tablet replica
I20260619 12:30:02.460954 2733 raft_consensus.cc:2229] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:02.461064 2733 raft_consensus.cc:2258] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:02.461187 2140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 12:30:02.461297 2734 ts_tablet_manager.cc:1905] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:02.461297 2732 ts_tablet_manager.cc:1905] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 12:30:02.461721 2140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:30:02.463783 2734 ts_tablet_manager.cc:1918] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:30:02.463773 2732 ts_tablet_manager.cc:1918] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:30:02.463857 2734 log.cc:1196] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/4d40a225010049b69e0c0228fd74bec8
I20260619 12:30:02.463857 2732 log.cc:1196] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-0/wal/wals/4d40a225010049b69e0c0228fd74bec8
I20260619 12:30:02.464113 2732 ts_tablet_manager.cc:1939] T 4d40a225010049b69e0c0228fd74bec8 P 3c6534fa12d84b19ab50ca02489beb17: Deleting consensus metadata
I20260619 12:30:02.464113 2734 ts_tablet_manager.cc:1939] T 4d40a225010049b69e0c0228fd74bec8 P eb6f7eb3e75b4d74abb614758f672e61: Deleting consensus metadata
I20260619 12:30:02.465003 2128 catalog_manager.cc:4841] TS 3c6534fa12d84b19ab50ca02489beb17 (127.0.98.1:42197): tablet 4d40a225010049b69e0c0228fd74bec8 (table test-table [id=888c0f79d1304c0a84fc6fe9a1b686be]) successfully deleted
I20260619 12:30:02.465124 2125 catalog_manager.cc:4841] TS eb6f7eb3e75b4d74abb614758f672e61 (127.0.98.2:42313): tablet 4d40a225010049b69e0c0228fd74bec8 (table test-table [id=888c0f79d1304c0a84fc6fe9a1b686be]) successfully deleted
I20260619 12:30:02.465543 2733 ts_tablet_manager.cc:1905] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:02.467669 2733 ts_tablet_manager.cc:1918] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:30:02.467730 2733 log.cc:1196] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/4d40a225010049b69e0c0228fd74bec8
I20260619 12:30:02.467931 2733 ts_tablet_manager.cc:1939] T 4d40a225010049b69e0c0228fd74bec8 P 9db27fd360884e7e92427fe6a322c61f: Deleting consensus metadata
I20260619 12:30:02.468611 2128 catalog_manager.cc:4841] TS 9db27fd360884e7e92427fe6a322c61f (127.0.98.3:36531): tablet 4d40a225010049b69e0c0228fd74bec8 (table test-table [id=888c0f79d1304c0a84fc6fe9a1b686be]) successfully deleted
I20260619 12:30:02.471567 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 3bbfbdc4de444aceb2dc298329fb9bbd (DEFAULT_TABLE table=test-table [id=24693a1de2fe4ef7936a0c23cf2e2673]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:02.471570 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 3bbfbdc4de444aceb2dc298329fb9bbd (DEFAULT_TABLE table=test-table [id=24693a1de2fe4ef7936a0c23cf2e2673]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:02.471627 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 3bbfbdc4de444aceb2dc298329fb9bbd (DEFAULT_TABLE table=test-table [id=24693a1de2fe4ef7936a0c23cf2e2673]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:02.471859 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bbfbdc4de444aceb2dc298329fb9bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:02.471856 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bbfbdc4de444aceb2dc298329fb9bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:02.471863 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bbfbdc4de444aceb2dc298329fb9bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:02.473922 2745 tablet_bootstrap.cc:492] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17: Bootstrap starting.
I20260619 12:30:02.474210 2743 tablet_bootstrap.cc:492] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61: Bootstrap starting.
I20260619 12:30:02.474838 2745 tablet_bootstrap.cc:654] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:02.475126 2743 tablet_bootstrap.cc:654] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:02.475791 2744 tablet_bootstrap.cc:492] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f: Bootstrap starting.
I20260619 12:30:02.476027 2745 tablet_bootstrap.cc:492] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17: No bootstrap required, opened a new log
I20260619 12:30:02.476128 2745 ts_tablet_manager.cc:1397] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 12:30:02.476318 2743 tablet_bootstrap.cc:492] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61: No bootstrap required, opened a new log
I20260619 12:30:02.476419 2743 ts_tablet_manager.cc:1397] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:30:02.476548 2745 raft_consensus.cc:348] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:02.476639 2744 tablet_bootstrap.cc:654] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:02.476675 2745 raft_consensus.cc:374] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:02.476717 2745 raft_consensus.cc:729] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c6534fa12d84b19ab50ca02489beb17, State: Initialized, Role: FOLLOWER
I20260619 12:30:02.476851 2745 consensus_queue.cc:260] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:02.476888 2743 raft_consensus.cc:348] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:02.477006 2743 raft_consensus.cc:374] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:02.477043 2743 raft_consensus.cc:729] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6f7eb3e75b4d74abb614758f672e61, State: Initialized, Role: FOLLOWER
I20260619 12:30:02.477154 2745 ts_tablet_manager.cc:1428] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 12:30:02.477186 2743 consensus_queue.cc:260] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:02.477502 2743 ts_tablet_manager.cc:1428] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 12:30:02.477947 2744 tablet_bootstrap.cc:492] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f: No bootstrap required, opened a new log
I20260619 12:30:02.478034 2744 ts_tablet_manager.cc:1397] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 12:30:02.478417 2744 raft_consensus.cc:348] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:02.478533 2744 raft_consensus.cc:374] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:02.478605 2744 raft_consensus.cc:729] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db27fd360884e7e92427fe6a322c61f, State: Initialized, Role: FOLLOWER
I20260619 12:30:02.478757 2744 consensus_queue.cc:260] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:02.479043 2744 ts_tablet_manager.cc:1428] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 12:30:02.512480 2726 raft_consensus.cc:482] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:30:02.512614 2726 raft_consensus.cc:504] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:02.513053 2726 leader_election.cc:288] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6f7eb3e75b4d74abb614758f672e61 (127.0.98.2:42313), 9db27fd360884e7e92427fe6a322c61f (127.0.98.3:36531)
I20260619 12:30:02.513669 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bbfbdc4de444aceb2dc298329fb9bbd" candidate_uuid: "3c6534fa12d84b19ab50ca02489beb17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db27fd360884e7e92427fe6a322c61f" is_pre_election: true
I20260619 12:30:02.513662 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bbfbdc4de444aceb2dc298329fb9bbd" candidate_uuid: "3c6534fa12d84b19ab50ca02489beb17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6f7eb3e75b4d74abb614758f672e61" is_pre_election: true
I20260619 12:30:02.513826 2437 raft_consensus.cc:2454] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c6534fa12d84b19ab50ca02489beb17 in term 0.
I20260619 12:30:02.513826 2571 raft_consensus.cc:2454] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c6534fa12d84b19ab50ca02489beb17 in term 0.
I20260619 12:30:02.514263 2229 leader_election.cc:302] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6534fa12d84b19ab50ca02489beb17, eb6f7eb3e75b4d74abb614758f672e61; no voters:
I20260619 12:30:02.514446 2726 raft_consensus.cc:2781] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:30:02.514513 2726 raft_consensus.cc:482] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:30:02.514544 2726 raft_consensus.cc:3037] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:02.515646 2726 raft_consensus.cc:504] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:02.515995 2726 leader_election.cc:288] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 [CANDIDATE]: Term 1 election: Requested vote from peers eb6f7eb3e75b4d74abb614758f672e61 (127.0.98.2:42313), 9db27fd360884e7e92427fe6a322c61f (127.0.98.3:36531)
I20260619 12:30:02.516381 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bbfbdc4de444aceb2dc298329fb9bbd" candidate_uuid: "3c6534fa12d84b19ab50ca02489beb17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6f7eb3e75b4d74abb614758f672e61"
I20260619 12:30:02.516528 2437 raft_consensus.cc:3037] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:02.516503 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bbfbdc4de444aceb2dc298329fb9bbd" candidate_uuid: "3c6534fa12d84b19ab50ca02489beb17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db27fd360884e7e92427fe6a322c61f"
I20260619 12:30:02.516642 2571 raft_consensus.cc:3037] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:02.517549 2437 raft_consensus.cc:2454] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c6534fa12d84b19ab50ca02489beb17 in term 1.
I20260619 12:30:02.517728 2571 raft_consensus.cc:2454] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c6534fa12d84b19ab50ca02489beb17 in term 1.
I20260619 12:30:02.517899 2229 leader_election.cc:302] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6534fa12d84b19ab50ca02489beb17, eb6f7eb3e75b4d74abb614758f672e61; no voters:
I20260619 12:30:02.518096 2726 raft_consensus.cc:2781] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:02.518198 2726 raft_consensus.cc:686] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 [term 1 LEADER]: Becoming Leader. State: Replica: 3c6534fa12d84b19ab50ca02489beb17, State: Running, Role: LEADER
I20260619 12:30:02.518338 2726 consensus_queue.cc:237] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } }
I20260619 12:30:02.519929 2140 catalog_manager.cc:5495] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c6534fa12d84b19ab50ca02489beb17 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3c6534fa12d84b19ab50ca02489beb17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c6534fa12d84b19ab50ca02489beb17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 } health_report { overall_health: UNKNOWN } } }
I20260619 12:30:02.546890 2437 raft_consensus.cc:1261] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61 [term 1 FOLLOWER]: Refusing update from remote peer 3c6534fa12d84b19ab50ca02489beb17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:02.546887 2571 raft_consensus.cc:1261] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f [term 1 FOLLOWER]: Refusing update from remote peer 3c6534fa12d84b19ab50ca02489beb17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:02.547254 2708 consensus_queue.cc:1035] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db27fd360884e7e92427fe6a322c61f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:02.547525 2708 consensus_queue.cc:1035] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6f7eb3e75b4d74abb614758f672e61" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 12:30:02.560909 2715 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 12:30:04.111475 2140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:04.111668 2140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:04.114100 2140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90d3391cea2c45c3b7ceb7f76e5c6abe: Sending DeleteTablet for 3 replicas of tablet 3bbfbdc4de444aceb2dc298329fb9bbd
I20260619 12:30:04.114630 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bbfbdc4de444aceb2dc298329fb9bbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43324
I20260619 12:30:04.114744 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bbfbdc4de444aceb2dc298329fb9bbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52692
I20260619 12:30:04.114960 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bbfbdc4de444aceb2dc298329fb9bbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59096
I20260619 12:30:04.115017 2783 tablet_replica.cc:330] T 3bbfbdc4de444aceb2dc298329fb9bbd P 3c6534fa12d84b19ab50ca02489beb17: stopping tablet replica
I20260619 12:30:04.115121 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 2213
I20260619 12:30:04.116532 2785 tablet_replica.cc:330] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f: stopping tablet replica
I20260619 12:30:04.116683 2785 raft_consensus.cc:2229] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:04.116812 2785 raft_consensus.cc:2258] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:04.117074 2784 tablet_replica.cc:330] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61: stopping tablet replica
I20260619 12:30:04.117236 2784 raft_consensus.cc:2229] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:04.117367 2784 raft_consensus.cc:2258] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:04.117939 2785 ts_tablet_manager.cc:1905] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:04.117938 2784 ts_tablet_manager.cc:1905] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:04.120545 2785 ts_tablet_manager.cc:1918] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 12:30:04.120554 2784 ts_tablet_manager.cc:1918] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 12:30:04.120630 2785 log.cc:1196] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/3bbfbdc4de444aceb2dc298329fb9bbd
I20260619 12:30:04.120630 2784 log.cc:1196] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/3bbfbdc4de444aceb2dc298329fb9bbd
I20260619 12:30:04.120903 2785 ts_tablet_manager.cc:1939] T 3bbfbdc4de444aceb2dc298329fb9bbd P 9db27fd360884e7e92427fe6a322c61f: Deleting consensus metadata
I20260619 12:30:04.120905 2784 ts_tablet_manager.cc:1939] T 3bbfbdc4de444aceb2dc298329fb9bbd P eb6f7eb3e75b4d74abb614758f672e61: Deleting consensus metadata
I20260619 12:30:04.121618 2128 catalog_manager.cc:4841] TS 9db27fd360884e7e92427fe6a322c61f (127.0.98.3:36531): tablet 3bbfbdc4de444aceb2dc298329fb9bbd (table test-table [id=24693a1de2fe4ef7936a0c23cf2e2673]) successfully deleted
I20260619 12:30:04.121737 2125 catalog_manager.cc:4841] TS eb6f7eb3e75b4d74abb614758f672e61 (127.0.98.2:42313): tablet 3bbfbdc4de444aceb2dc298329fb9bbd (table test-table [id=24693a1de2fe4ef7936a0c23cf2e2673]) successfully deleted
W20260619 12:30:04.122908 2128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42197 (error 108)
I20260619 12:30:04.123159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 2353
I20260619 12:30:04.124749 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:04.124262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42197 (local address 127.0.0.1:43416)
0619 12:30:04.124529 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:04.124619 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
W20260619 12:30:04.124864 2128 catalog_manager.cc:4584] TS 3c6534fa12d84b19ab50ca02489beb17 (127.0.98.1:42197): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3bbfbdc4de444aceb2dc298329fb9bbd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42197: connect: Connection refused (error 111)
I20260619 12:30:04.129778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 2493
I20260619 12:30:04.136921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 2109
2026-06-19T12:30:04Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6696 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:30:04 dist-test-slave-fg3h krb5kdc[2790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:30:04 dist-test-slave-fg3h krb5kdc[2790](info): set up 2 sockets
Jun 19 12:30:04 dist-test-slave-fg3h krb5kdc[2790](info): commencing operation
krb5kdc: starting...
W20260619 12:30:05.199508 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:30:05 dist-test-slave-fg3h krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:30:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:30:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:30:05.365865 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41813
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:05.475870 2803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:05.476133 2803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:05.476174 2803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:05.480175 2803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:05.480243 2803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:05.480263 2803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:05.480280 2803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:05.480296 2803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:30:05.485554 2803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:05.486527 2803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:05.487475 2803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:05.495780 2808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:05.495777 2809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:05.495846 2811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:05.496034 2803 server_base.cc:1034] running on GCE node
I20260619 12:30:05.496701 2803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:05.497874 2803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:05.499061 2803 hybrid_clock.cc:648] HybridClock initialized: now 1781872205499047 us; error 26 us; skew 500 ppm
Jun 19 12:30:05 dist-test-slave-fg3h krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:05.501931 2803 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:30:05.502764 2803 webserver.cc:460] Webserver started at http://127.0.98.62:44505/ using document root <none> and password file <none>
I20260619 12:30:05.503289 2803 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:05.503348 2803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:05.503513 2803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:05.505560 2803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "f30f21f424484a6aae67bc6b1a861b56"
format_stamp: "Formatted at 2026-06-19 12:30:05 on dist-test-slave-fg3h"
server_key: "21b23d77667c3cfae2472dd96e74cd51"
server_key_iv: "cb60a0891a18a6c2625ea08336e89004"
server_key_version: "encryptionkey@0"
I20260619 12:30:05.506059 2803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "f30f21f424484a6aae67bc6b1a861b56"
format_stamp: "Formatted at 2026-06-19 12:30:05 on dist-test-slave-fg3h"
server_key: "21b23d77667c3cfae2472dd96e74cd51"
server_key_iv: "cb60a0891a18a6c2625ea08336e89004"
server_key_version: "encryptionkey@0"
I20260619 12:30:05.509681 2803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:30:05.511953 2818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:05.513341 2803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:05.513473 2803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "f30f21f424484a6aae67bc6b1a861b56"
format_stamp: "Formatted at 2026-06-19 12:30:05 on dist-test-slave-fg3h"
server_key: "21b23d77667c3cfae2472dd96e74cd51"
server_key_iv: "cb60a0891a18a6c2625ea08336e89004"
server_key_version: "encryptionkey@0"
I20260619 12:30:05.513574 2803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:05.528404 2803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:05.531891 2803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:05.532095 2803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:05.539567 2803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41813
I20260619 12:30:05.539592 2870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41813 every 8 connection(s)
I20260619 12:30:05.540561 2803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:30:05.541913 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 2803
I20260619 12:30:05.542002 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:30:05.542297 392 external_mini_cluster.cc:1428] Setting key 0b98175d4c5616d0c86d07f3445ee77b
I20260619 12:30:05.543748 2871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:05.549366 2871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56: Bootstrap starting.
Jun 19 12:30:05 dist-test-slave-fg3h krb5kdc[2790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:05.552038 2871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:05.552778 2871 log.cc:824] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:05.554960 2871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56: No bootstrap required, opened a new log
I20260619 12:30:05.556614 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:05.543712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.0.98.62:41813)
0619 12:30:05.544160 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:05.544176 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:05.544212 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:30:05.544876 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:05.544893 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:05.545295 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:05.545643 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:05.546444 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:05.547459 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:05.548322 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:05.548782 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:05.548944 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:05.551697 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:05.551703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:05.551715 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:05.551742 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:05.553737 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:05.554159 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:05.554164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:05.554170 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:05.554280 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:05.554563 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:05.554567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:05.554571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:05.555007 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:05.555198 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:05.555538 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 12:30:05.555866 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":140,"threads_started":1}
I20260619 12:30:05.558957 2871 raft_consensus.cc:348] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30f21f424484a6aae67bc6b1a861b56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41813 } }
I20260619 12:30:05.559294 2871 raft_consensus.cc:374] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:05.559361 2871 raft_consensus.cc:729] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f30f21f424484a6aae67bc6b1a861b56, State: Initialized, Role: FOLLOWER
I20260619 12:30:05.559870 2871 consensus_queue.cc:260] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30f21f424484a6aae67bc6b1a861b56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41813 } }
I20260619 12:30:05.560040 2871 raft_consensus.cc:388] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:30:05.560113 2871 raft_consensus.cc:482] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:30:05.560216 2871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:05.561605 2871 raft_consensus.cc:504] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30f21f424484a6aae67bc6b1a861b56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41813 } }
I20260619 12:30:05.562017 2871 leader_election.cc:302] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f30f21f424484a6aae67bc6b1a861b56; no voters:
I20260619 12:30:05.562376 2871 leader_election.cc:288] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:30:05.562460 2876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:05.562651 2876 raft_consensus.cc:686] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56 [term 1 LEADER]: Becoming Leader. State: Replica: f30f21f424484a6aae67bc6b1a861b56, State: Running, Role: LEADER
I20260619 12:30:05.562935 2876 consensus_queue.cc:237] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30f21f424484a6aae67bc6b1a861b56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41813 } }
I20260619 12:30:05.563647 2871 sys_catalog.cc:564] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:05.564544 2878 sys_catalog.cc:455] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f30f21f424484a6aae67bc6b1a861b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30f21f424484a6aae67bc6b1a861b56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41813 } } }
I20260619 12:30:05.564769 2878 sys_catalog.cc:458] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:05.565141 2877 sys_catalog.cc:455] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f30f21f424484a6aae67bc6b1a861b56. Latest consensus state: current_term: 1 leader_uuid: "f30f21f424484a6aae67bc6b1a861b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30f21f424484a6aae67bc6b1a861b56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41813 } } }
I20260619 12:30:05.565292 2877 sys_catalog.cc:458] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:05.565639 2885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:30:05.569310 2885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:30:05.574632 2885 catalog_manager.cc:1347] Generated new cluster ID: e7b1b25f374a4cceb4381c0bba3823b7
I20260619 12:30:05.574700 2885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:30:05.581575 2885 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:30:05.582440 2885 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:30:05.597909 2885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f30f21f424484a6aae67bc6b1a861b56: Generated new TSK 0
I20260619 12:30:05.598645 2885 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:30:05.626413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41813
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:05.736347 2897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:05.736608 2897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:05.736651 2897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:05.740682 2897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:05.740753 2897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:05.740811 2897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:30:05.746184 2897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:05.747251 2897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:05.748261 2897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:05.758111 2903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:05.758095 2902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:05.758230 2897 server_base.cc:1034] running on GCE node
W20260619 12:30:05.758090 2905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:05.758656 2897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:05.759315 2897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:05.760504 2897 hybrid_clock.cc:648] HybridClock initialized: now 1781872205760485 us; error 42 us; skew 500 ppm
Jun 19 12:30:05 dist-test-slave-fg3h krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:05.763355 2897 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:30:05.764395 2897 webserver.cc:460] Webserver started at http://127.0.98.1:40057/ using document root <none> and password file <none>
I20260619 12:30:05.765038 2897 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:05.765153 2897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:05.765388 2897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:05.767503 2897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "3917c5ee393049f487205eb1fab76fcf"
format_stamp: "Formatted at 2026-06-19 12:30:05 on dist-test-slave-fg3h"
server_key: "e32a9214889c88e077d9d5413a68b4d7"
server_key_iv: "060eeafd643ebaa68c5429e25dd32d2e"
server_key_version: "encryptionkey@0"
I20260619 12:30:05.768070 2897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3917c5ee393049f487205eb1fab76fcf"
format_stamp: "Formatted at 2026-06-19 12:30:05 on dist-test-slave-fg3h"
server_key: "e32a9214889c88e077d9d5413a68b4d7"
server_key_iv: "060eeafd643ebaa68c5429e25dd32d2e"
server_key_version: "encryptionkey@0"
I20260619 12:30:05.771752 2897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 12:30:05.774178 2912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:05.775468 2897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:30:05.775609 2897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "3917c5ee393049f487205eb1fab76fcf"
format_stamp: "Formatted at 2026-06-19 12:30:05 on dist-test-slave-fg3h"
server_key: "e32a9214889c88e077d9d5413a68b4d7"
server_key_iv: "060eeafd643ebaa68c5429e25dd32d2e"
server_key_version: "encryptionkey@0"
I20260619 12:30:05.775719 2897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:05.802002 2897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:05.805399 2897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:05.805598 2897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:05.806185 2897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:05.806995 2897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:05.807062 2897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:05.807129 2897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:05.807174 2897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:05.815646 2897 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34727
I20260619 12:30:05.815668 3025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34727 every 8 connection(s)
I20260619 12:30:05.816577 2897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:30:05.822646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 2897
I20260619 12:30:05.822763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:30:05.823036 392 external_mini_cluster.cc:1428] Setting key c900b83ea2b6a2ca5df3ff6b10429efd
Jun 19 12:30:05 dist-test-slave-fg3h krb5kdc[2790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:05.830410 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:05.818385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41889 (local address 127.0.98.62:41813)
0619 12:30:05.818507 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:05.818510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:05.819308 (+ 798us) server_negotiation.cc:407] Connection header received
0619 12:30:05.820688 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:05.820692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:05.820747 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:05.820848 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:05.822042 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:05.822704 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:05.824031 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:05.824288 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:05.824423 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:05.826857 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:05.826864 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:05.826871 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:05.826906 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:05.828336 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:05.828823 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:05.828836 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:05.828838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:05.828917 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:05.829299 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:05.829303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:05.829305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:05.829486 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:05.829587 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:05.830112 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 12:30:05.830237 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 12:30:05.831202 3028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:05.818718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41813 (local address 127.0.98.1:41889)
0619 12:30:05.819162 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:05.819203 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:05.820479 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:05.821016 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:05.821023 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:05.821438 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:05.821903 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:05.821917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:05.822844 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:05.822852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:05.823911 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:05.823921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:05.824473 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:05.824476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:05.824609 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:05.825237 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:05.825253 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:05.826689 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:05.828451 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:05.828457 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:05.828472 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:05.828721 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:05.829008 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:05.829012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:05.829014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:05.829201 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:05.829595 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:05.829604 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:05.829842 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 12:30:05.830054 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 12:30:05.830380 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":138,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:30:05.832558 3026 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41813
I20260619 12:30:05.832810 3026 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:05.833453 3026 heartbeater.cc:511] Master 127.0.98.62:41813 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:30:05.835281 2835 ts_manager.cc:194] Registered new tserver with Master: 3917c5ee393049f487205eb1fab76fcf (127.0.98.1:34727)
I20260619 12:30:05.836459 2835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:30:05.852672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41813
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:05.962555 3031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:05.962832 3031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:05.962899 3031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:05.967224 3031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:05.967310 3031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:05.967406 3031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:30:05.972594 3031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:05.973722 3031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:05.974735 3031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:05.984037 3036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:05.984164 3037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:05.984037 3039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:05.984187 3031 server_base.cc:1034] running on GCE node
I20260619 12:30:05.984759 3031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:05.985445 3031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:05.986599 3031 hybrid_clock.cc:648] HybridClock initialized: now 1781872205986582 us; error 31 us; skew 500 ppm
Jun 19 12:30:05 dist-test-slave-fg3h krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:05.989368 3031 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:30:05.990206 3031 webserver.cc:460] Webserver started at http://127.0.98.2:41471/ using document root <none> and password file <none>
I20260619 12:30:05.990753 3031 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:05.990813 3031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:05.991035 3031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:05.993026 3031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "a6c6892c38224551b7c96ebac0d88e42"
format_stamp: "Formatted at 2026-06-19 12:30:05 on dist-test-slave-fg3h"
server_key: "ca3c91c9585f5ee2551d0f45e80f7c40"
server_key_iv: "b2d41db4398b8afa2b6a31666cab0fc7"
server_key_version: "encryptionkey@0"
I20260619 12:30:05.993631 3031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a6c6892c38224551b7c96ebac0d88e42"
format_stamp: "Formatted at 2026-06-19 12:30:05 on dist-test-slave-fg3h"
server_key: "ca3c91c9585f5ee2551d0f45e80f7c40"
server_key_iv: "b2d41db4398b8afa2b6a31666cab0fc7"
server_key_version: "encryptionkey@0"
I20260619 12:30:05.997346 3031 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:30:05.999858 3046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:06.001210 3031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:06.001332 3031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "a6c6892c38224551b7c96ebac0d88e42"
format_stamp: "Formatted at 2026-06-19 12:30:05 on dist-test-slave-fg3h"
server_key: "ca3c91c9585f5ee2551d0f45e80f7c40"
server_key_iv: "b2d41db4398b8afa2b6a31666cab0fc7"
server_key_version: "encryptionkey@0"
I20260619 12:30:06.001457 3031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:06.019821 3031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:06.022971 3031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:06.023180 3031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:06.023984 3031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:06.025009 3031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:06.025125 3031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:06.025197 3031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:06.025231 3031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:06.034534 3031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46715
I20260619 12:30:06.034586 3159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46715 every 8 connection(s)
I20260619 12:30:06.035444 3031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:30:06.039204 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 3031
I20260619 12:30:06.039350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:30:06.039656 392 external_mini_cluster.cc:1428] Setting key e016bbe3727574c87f37256fc225566a
Jun 19 12:30:06 dist-test-slave-fg3h krb5kdc[2790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:30:06.051493 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:06.037301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47633 (local address 127.0.98.62:41813)
0619 12:30:06.037435 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:06.037439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:06.038316 (+ 877us) server_negotiation.cc:407] Connection header received
0619 12:30:06.040107 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:06.040110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:06.040167 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:06.040271 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:06.041783 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:06.042330 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:06.043639 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:06.044115 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:06.044258 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:06.047151 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:06.047158 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:06.047163 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:06.047203 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:06.049364 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:06.049899 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:06.049904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:06.049906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:06.049999 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:06.050391 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:06.050399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:06.050403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:06.050646 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:06.050733 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:06.051238 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 12:30:06.051365 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 12:30:06.052667 3162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:06.037678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41813 (local address 127.0.98.2:47633)
0619 12:30:06.038153 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:06.038195 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:06.039886 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:06.040447 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:06.040456 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:06.040935 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:06.041608 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:06.041632 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:06.042464 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:06.042471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:06.043504 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:06.043513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:06.044266 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:06.044268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:06.044408 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:06.045053 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:06.045069 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:06.046968 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:06.049501 (+ 2533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:06.049505 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:06.049517 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:06.049776 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:06.050094 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:06.050096 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:06.050097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:06.050281 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:06.050737 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:06.050742 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:06.050973 (+ 231us) client_negotiation.cc:769] Sending connection context
0619 12:30:06.051150 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 12:30:06.051468 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":121,"threads_started":1}
I20260619 12:30:06.054024 3160 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41813
I20260619 12:30:06.054292 3160 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:06.054781 3160 heartbeater.cc:511] Master 127.0.98.62:41813 requested a full tablet report, sending...
I20260619 12:30:06.055835 2835 ts_manager.cc:194] Registered new tserver with Master: a6c6892c38224551b7c96ebac0d88e42 (127.0.98.2:46715)
I20260619 12:30:06.056623 2835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:30:06.068578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41813
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:06.179083 3165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:06.179374 3165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:06.179445 3165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:06.183786 3165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:06.183880 3165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:06.183967 3165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:30:06.189257 3165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:06.190327 3165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:06.191366 3165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:06.200793 3173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:06.200795 3171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:06.200795 3170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:06.201068 3165 server_base.cc:1034] running on GCE node
I20260619 12:30:06.201615 3165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:06.202241 3165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:06.203428 3165 hybrid_clock.cc:648] HybridClock initialized: now 1781872206203407 us; error 32 us; skew 500 ppm
Jun 19 12:30:06 dist-test-slave-fg3h krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:06.206251 3165 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:30:06.207113 3165 webserver.cc:460] Webserver started at http://127.0.98.3:38513/ using document root <none> and password file <none>
I20260619 12:30:06.207621 3165 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:06.207679 3165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:06.207836 3165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:06.209841 3165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "06704639884d4d5da065677073d7af18"
format_stamp: "Formatted at 2026-06-19 12:30:06 on dist-test-slave-fg3h"
server_key: "e57b07c53bb66aa4e95ffc29f7cbf4b9"
server_key_iv: "b7d4cfb5c07f3ac822ff204ef2303f7b"
server_key_version: "encryptionkey@0"
I20260619 12:30:06.210345 3165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "06704639884d4d5da065677073d7af18"
format_stamp: "Formatted at 2026-06-19 12:30:06 on dist-test-slave-fg3h"
server_key: "e57b07c53bb66aa4e95ffc29f7cbf4b9"
server_key_iv: "b7d4cfb5c07f3ac822ff204ef2303f7b"
server_key_version: "encryptionkey@0"
I20260619 12:30:06.213899 3165 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:30:06.216203 3180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:06.217620 3165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:06.217746 3165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "06704639884d4d5da065677073d7af18"
format_stamp: "Formatted at 2026-06-19 12:30:06 on dist-test-slave-fg3h"
server_key: "e57b07c53bb66aa4e95ffc29f7cbf4b9"
server_key_iv: "b7d4cfb5c07f3ac822ff204ef2303f7b"
server_key_version: "encryptionkey@0"
I20260619 12:30:06.217865 3165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:06.239763 3165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:06.243443 3165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:06.243660 3165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:06.244318 3165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:06.245204 3165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:06.245273 3165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:06.245337 3165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:06.245376 3165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:06.254578 3165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35133
I20260619 12:30:06.254594 3293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35133 every 8 connection(s)
I20260619 12:30:06.255517 3165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:30:06.264875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 3165
I20260619 12:30:06.265065 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:30:06.265398 392 external_mini_cluster.cc:1428] Setting key cf512def119c408ec375d603dde1de93
Jun 19 12:30:06 dist-test-slave-fg3h krb5kdc[2790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:06.270229 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:06.257447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38373 (local address 127.0.98.62:41813)
0619 12:30:06.257577 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:06.257580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:06.258440 (+ 860us) server_negotiation.cc:407] Connection header received
0619 12:30:06.259900 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:06.259903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:06.259959 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:06.260071 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:06.261320 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:06.261852 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:06.263128 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:06.263344 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:06.263451 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:06.266458 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:06.266462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:06.266465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:06.266493 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:06.268070 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:06.268628 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:06.268633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:06.268635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:06.268719 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:06.269064 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:06.269068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:06.269070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:06.269287 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:06.269392 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:06.269939 (+ 547us) server_negotiation.cc:299] Negotiation successful
0619 12:30:06.270063 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 12:30:06.271124 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:06.257795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41813 (local address 127.0.98.3:38373)
0619 12:30:06.258273 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:06.258316 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:06.259675 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:06.260249 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:06.260256 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:06.260632 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:06.261178 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:06.261195 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:06.261980 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:06.261983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:06.262985 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:06.262994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:06.263494 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:06.263499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:06.263737 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:06.264384 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:06.264400 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:06.266252 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:06.268207 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:06.268213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:06.268229 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:06.268523 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:06.268819 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:06.268822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:06.268824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:06.268969 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:06.269410 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:06.269417 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:06.269664 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 12:30:06.269878 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 12:30:06.270215 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":124,"threads_started":1}
I20260619 12:30:06.272269 3294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41813
I20260619 12:30:06.272527 3294 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:06.273025 3294 heartbeater.cc:511] Master 127.0.98.62:41813 requested a full tablet report, sending...
I20260619 12:30:06.274142 2835 ts_manager.cc:194] Registered new tserver with Master: 06704639884d4d5da065677073d7af18 (127.0.98.3:35133)
I20260619 12:30:06.274734 2835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38373
I20260619 12:30:06.279484 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 12:30:06.292120 2874 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:06.290615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.0.98.62:41813)
0619 12:30:06.290875 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:06.290880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:06.291047 (+ 167us) server_negotiation.cc:407] Connection header received
0619 12:30:06.291159 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:06.291162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:06.291204 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:06.291277 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:06.291981 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37556: BlockingRecv error: recv got EOF from 127.0.0.1:37556 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
I20260619 12:30:06.293854 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 2897
I20260619 12:30:06.299327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 3031
I20260619 12:30:06.304517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 3165
I20260619 12:30:06.309945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 2803
2026-06-19T12:30:06Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2171 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:30:06 dist-test-slave-fg3h krb5kdc[3306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:30:06 dist-test-slave-fg3h krb5kdc[3306](info): set up 2 sockets
Jun 19 12:30:06 dist-test-slave-fg3h krb5kdc[3306](info): commencing operation
krb5kdc: starting...
W20260619 12:30:07.345223 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:30:07 dist-test-slave-fg3h krb5kdc[3306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:30:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:30:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:30:07.468364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42465
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:07.583545 3319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:07.583822 3319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:07.583894 3319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:07.588245 3319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:07.588315 3319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:07.588335 3319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:07.588351 3319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:07.588366 3319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:30:07.593717 3319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:07.594806 3319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:07.595815 3319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:07.604116 3327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:07.604195 3324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:07.604161 3325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:07.604337 3319 server_base.cc:1034] running on GCE node
I20260619 12:30:07.605002 3319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:07.606148 3319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:07.607339 3319 hybrid_clock.cc:648] HybridClock initialized: now 1781872207607305 us; error 47 us; skew 500 ppm
Jun 19 12:30:07 dist-test-slave-fg3h krb5kdc[3306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:07.615100 3319 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:30:07.616062 3319 webserver.cc:460] Webserver started at http://127.0.98.62:39251/ using document root <none> and password file <none>
I20260619 12:30:07.616662 3319 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:07.616756 3319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:07.616987 3319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:07.619216 3319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "dff45a8110cd4aaebfdc1f7dc2a4565a"
format_stamp: "Formatted at 2026-06-19 12:30:07 on dist-test-slave-fg3h"
server_key: "00b9627eda16c64a954847c36973a62a"
server_key_iv: "1873f81f026582532af02fab9e903605"
server_key_version: "encryptionkey@0"
I20260619 12:30:07.619787 3319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "dff45a8110cd4aaebfdc1f7dc2a4565a"
format_stamp: "Formatted at 2026-06-19 12:30:07 on dist-test-slave-fg3h"
server_key: "00b9627eda16c64a954847c36973a62a"
server_key_iv: "1873f81f026582532af02fab9e903605"
server_key_version: "encryptionkey@0"
I20260619 12:30:07.623617 3319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:30:07.626343 3334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:07.627588 3319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:30:07.627729 3319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "dff45a8110cd4aaebfdc1f7dc2a4565a"
format_stamp: "Formatted at 2026-06-19 12:30:07 on dist-test-slave-fg3h"
server_key: "00b9627eda16c64a954847c36973a62a"
server_key_iv: "1873f81f026582532af02fab9e903605"
server_key_version: "encryptionkey@0"
I20260619 12:30:07.627846 3319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:07.654371 3319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:07.657451 3319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:07.657661 3319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:07.664788 3319 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42465
I20260619 12:30:07.665163 3386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42465 every 8 connection(s)
I20260619 12:30:07.665838 3319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:30:07.668606 3387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:07.674001 3387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a: Bootstrap starting.
I20260619 12:30:07.674897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 3319
I20260619 12:30:07.675019 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:30:07.675325 392 external_mini_cluster.cc:1428] Setting key 2a934854f03cec60bf626de943598c00
I20260619 12:30:07.676909 3387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:07.677690 3387 log.cc:824] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:07.679837 3387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a: No bootstrap required, opened a new log
Jun 19 12:30:07 dist-test-slave-fg3h krb5kdc[3306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:07.684423 3387 raft_consensus.cc:348] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff45a8110cd4aaebfdc1f7dc2a4565a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42465 } }
I20260619 12:30:07.684701 3387 raft_consensus.cc:374] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:07.684777 3387 raft_consensus.cc:729] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff45a8110cd4aaebfdc1f7dc2a4565a, State: Initialized, Role: FOLLOWER
I20260619 12:30:07.685335 3387 consensus_queue.cc:260] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff45a8110cd4aaebfdc1f7dc2a4565a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42465 } }
I20260619 12:30:07.685514 3387 raft_consensus.cc:388] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:30:07.685585 3387 raft_consensus.cc:482] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:30:07.685719 3387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:07.686612 3387 raft_consensus.cc:504] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff45a8110cd4aaebfdc1f7dc2a4565a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42465 } }
I20260619 12:30:07.686952 3387 leader_election.cc:302] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dff45a8110cd4aaebfdc1f7dc2a4565a; no voters:
I20260619 12:30:07.687302 3387 leader_election.cc:288] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:30:07.687372 3392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:07.687572 3392 raft_consensus.cc:686] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a [term 1 LEADER]: Becoming Leader. State: Replica: dff45a8110cd4aaebfdc1f7dc2a4565a, State: Running, Role: LEADER
I20260619 12:30:07.687917 3392 consensus_queue.cc:237] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff45a8110cd4aaebfdc1f7dc2a4565a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42465 } }
I20260619 12:30:07.688637 3387 sys_catalog.cc:564] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:07.689656 3393 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dff45a8110cd4aaebfdc1f7dc2a4565a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff45a8110cd4aaebfdc1f7dc2a4565a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42465 } } }
I20260619 12:30:07.689795 3393 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:07.690171 3401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:30:07.690142 3394 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dff45a8110cd4aaebfdc1f7dc2a4565a. Latest consensus state: current_term: 1 leader_uuid: "dff45a8110cd4aaebfdc1f7dc2a4565a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff45a8110cd4aaebfdc1f7dc2a4565a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42465 } } }
I20260619 12:30:07.690275 3394 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:07.690565 3390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:07.676442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60990 (local address 127.0.98.62:42465)
0619 12:30:07.676864 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:07.676878 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:07.676912 (+ 34us) server_negotiation.cc:407] Connection header received
0619 12:30:07.677622 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:07.677640 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:07.678002 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:07.678387 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:07.679256 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:07.680264 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:07.681462 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:07.681938 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:07.682070 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:07.685001 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:07.685009 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:07.685027 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:07.685059 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:07.687001 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:07.687503 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:07.687508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:07.687514 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:07.687685 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:07.687992 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:07.687995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:07.687997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:07.688415 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:07.688637 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:07.689111 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 12:30:07.689426 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":153,"threads_started":1}
I20260619 12:30:07.693580 3401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:30:07.693681 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:30:07.700466 3401 catalog_manager.cc:1347] Generated new cluster ID: 937ac279a786485e8f88a449564bb503
I20260619 12:30:07.700528 3401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:30:07.715773 3390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:07.710073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32770 (local address 127.0.98.62:42465)
0619 12:30:07.710188 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:07.710193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:07.710451 (+ 258us) server_negotiation.cc:407] Connection header received
0619 12:30:07.710567 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:07.710570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:07.710626 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:07.710718 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:07.711271 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:07.712096 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:07.713288 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:07.713672 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:07.713784 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:07.714712 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:07.714716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:07.714719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:07.715321 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:07.715355 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:07.715374 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:07.715479 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:07.715549 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 12:30:07.715613 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":44}
I20260619 12:30:07.718801 3401 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:30:07.719872 3401 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:30:07.729830 3401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dff45a8110cd4aaebfdc1f7dc2a4565a: Generated new TSK 0
I20260619 12:30:07.730521 3401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:30:07.737075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 3319
2026-06-19T12:30:07Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1424 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:30:07 dist-test-slave-fg3h krb5kdc[3420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:30:07 dist-test-slave-fg3h krb5kdc[3420](info): set up 2 sockets
Jun 19 12:30:07 dist-test-slave-fg3h krb5kdc[3420](info): commencing operation
krb5kdc: starting...
W20260619 12:30:08.777468 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:30:08 dist-test-slave-fg3h krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:30:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:30:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:30:08.903836 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42623
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:09.012063 3433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:09.012367 3433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:09.012437 3433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:09.016475 3433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:09.016562 3433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:09.016603 3433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:09.016654 3433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:09.016690 3433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:30:09.021973 3433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:09.023021 3433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:09.024071 3433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:30:09.032836 3433 server_base.cc:1034] running on GCE node
W20260619 12:30:09.032825 3438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:09.032821 3441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:09.033028 3439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:09.033699 3433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:09.034842 3433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:09.036026 3433 hybrid_clock.cc:648] HybridClock initialized: now 1781872209036012 us; error 33 us; skew 500 ppm
Jun 19 12:30:09 dist-test-slave-fg3h krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:09.039579 3433 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:30:09.040549 3433 webserver.cc:460] Webserver started at http://127.0.98.62:45711/ using document root <none> and password file <none>
I20260619 12:30:09.041188 3433 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:09.041265 3433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:09.041450 3433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:09.043576 3433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "efcfdf7fdf2c4682bb85b75f45612357"
format_stamp: "Formatted at 2026-06-19 12:30:09 on dist-test-slave-fg3h"
server_key: "083720ceb1609010fad51ba49e5d7c9c"
server_key_iv: "8a2b275bb564b2f32b668726b46ebe1d"
server_key_version: "encryptionkey@0"
I20260619 12:30:09.044166 3433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "efcfdf7fdf2c4682bb85b75f45612357"
format_stamp: "Formatted at 2026-06-19 12:30:09 on dist-test-slave-fg3h"
server_key: "083720ceb1609010fad51ba49e5d7c9c"
server_key_iv: "8a2b275bb564b2f32b668726b46ebe1d"
server_key_version: "encryptionkey@0"
I20260619 12:30:09.048167 3433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 12:30:09.050812 3448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:09.052089 3433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:30:09.052237 3433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "efcfdf7fdf2c4682bb85b75f45612357"
format_stamp: "Formatted at 2026-06-19 12:30:09 on dist-test-slave-fg3h"
server_key: "083720ceb1609010fad51ba49e5d7c9c"
server_key_iv: "8a2b275bb564b2f32b668726b46ebe1d"
server_key_version: "encryptionkey@0"
I20260619 12:30:09.052395 3433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:09.064291 3433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:09.067847 3433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:09.068066 3433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:09.075654 3433 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42623
I20260619 12:30:09.075978 3500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42623 every 8 connection(s)
I20260619 12:30:09.076650 3433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:30:09.079500 3501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:09.079774 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 3433
I20260619 12:30:09.079905 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:30:09.080185 392 external_mini_cluster.cc:1428] Setting key 221d0ae49b4aba3ad0ff318eb47756b6
I20260619 12:30:09.086522 3501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357: Bootstrap starting.
I20260619 12:30:09.089319 3501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357: Neither blocks nor log segments found. Creating new log.
Jun 19 12:30:09 dist-test-slave-fg3h krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:09.090083 3501 log.cc:824] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:09.092167 3501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357: No bootstrap required, opened a new log
I20260619 12:30:09.094946 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:09.081680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.0.98.62:42623)
0619 12:30:09.082078 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:09.082095 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:09.082141 (+ 46us) server_negotiation.cc:407] Connection header received
0619 12:30:09.082821 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:09.082838 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:09.083165 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:09.083484 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:09.084395 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:09.085505 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:09.086722 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:09.087207 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:09.087338 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:09.090288 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:09.090293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:09.090307 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:09.090334 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:09.092120 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:09.092627 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:09.092637 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:09.092642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:09.092763 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:09.093036 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:09.093041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:09.093042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:09.093486 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:09.093674 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:09.093953 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 12:30:09.094236 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":134,"threads_started":1}
I20260619 12:30:09.095916 3501 raft_consensus.cc:348] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcfdf7fdf2c4682bb85b75f45612357" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42623 } }
I20260619 12:30:09.096185 3501 raft_consensus.cc:374] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:09.096241 3501 raft_consensus.cc:729] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efcfdf7fdf2c4682bb85b75f45612357, State: Initialized, Role: FOLLOWER
I20260619 12:30:09.096710 3501 consensus_queue.cc:260] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcfdf7fdf2c4682bb85b75f45612357" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42623 } }
I20260619 12:30:09.096866 3501 raft_consensus.cc:388] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:30:09.096921 3501 raft_consensus.cc:482] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:30:09.097015 3501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:09.098009 3501 raft_consensus.cc:504] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcfdf7fdf2c4682bb85b75f45612357" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42623 } }
I20260619 12:30:09.098308 3501 leader_election.cc:302] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efcfdf7fdf2c4682bb85b75f45612357; no voters:
I20260619 12:30:09.098574 3501 leader_election.cc:288] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:30:09.098702 3506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:09.099058 3506 raft_consensus.cc:686] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357 [term 1 LEADER]: Becoming Leader. State: Replica: efcfdf7fdf2c4682bb85b75f45612357, State: Running, Role: LEADER
I20260619 12:30:09.099496 3506 consensus_queue.cc:237] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcfdf7fdf2c4682bb85b75f45612357" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42623 } }
I20260619 12:30:09.099864 3501 sys_catalog.cc:564] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:09.101548 3507 sys_catalog.cc:455] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efcfdf7fdf2c4682bb85b75f45612357" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcfdf7fdf2c4682bb85b75f45612357" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42623 } } }
I20260619 12:30:09.101679 3507 sys_catalog.cc:458] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:09.101994 3508 sys_catalog.cc:455] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efcfdf7fdf2c4682bb85b75f45612357. Latest consensus state: current_term: 1 leader_uuid: "efcfdf7fdf2c4682bb85b75f45612357" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcfdf7fdf2c4682bb85b75f45612357" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42623 } } }
I20260619 12:30:09.102624 3508 sys_catalog.cc:458] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357 [sys.catalog]: This master's current role is: LEADER
W20260619 12:30:09.104557 3521 catalog_manager.cc:1558] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:30:09.104620 3521 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:30:09.104719 3520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:30:09.105458 3520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:30:09.111380 3520 catalog_manager.cc:1347] Generated new cluster ID: cee6cfb5a95d4fe6aa216578366cd090
I20260619 12:30:09.111455 3520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:30:09.140228 3520 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:30:09.141248 3520 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:30:09.148825 3520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efcfdf7fdf2c4682bb85b75f45612357: Generated new TSK 0
I20260619 12:30:09.149501 3520 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:30:09.180756 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42623
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:09.291209 3527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:09.291486 3527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:09.291529 3527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:09.295560 3527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:09.295627 3527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:09.295687 3527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:30:09.301288 3527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:09.302359 3527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:09.303319 3527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:09.312960 3535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:09.312952 3533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:09.312979 3532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:09.313131 3527 server_base.cc:1034] running on GCE node
I20260619 12:30:09.313762 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:09.314419 3527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:09.315618 3527 hybrid_clock.cc:648] HybridClock initialized: now 1781872209315606 us; error 34 us; skew 500 ppm
Jun 19 12:30:09 dist-test-slave-fg3h krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:09.318710 3527 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:30:09.319770 3527 webserver.cc:460] Webserver started at http://127.0.98.1:42053/ using document root <none> and password file <none>
I20260619 12:30:09.320451 3527 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:09.320520 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:09.320688 3527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:09.322824 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "57cf7df1fa6840198c18bf54b5b1de69"
format_stamp: "Formatted at 2026-06-19 12:30:09 on dist-test-slave-fg3h"
server_key: "f61e85cfac4e97d14acb9e67e367a0e9"
server_key_iv: "f97296a25ad43b1067748a67eb2cc7e9"
server_key_version: "encryptionkey@0"
I20260619 12:30:09.323351 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "57cf7df1fa6840198c18bf54b5b1de69"
format_stamp: "Formatted at 2026-06-19 12:30:09 on dist-test-slave-fg3h"
server_key: "f61e85cfac4e97d14acb9e67e367a0e9"
server_key_iv: "f97296a25ad43b1067748a67eb2cc7e9"
server_key_version: "encryptionkey@0"
I20260619 12:30:09.327111 3527 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 12:30:09.329602 3542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:09.330840 3527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:09.330937 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "57cf7df1fa6840198c18bf54b5b1de69"
format_stamp: "Formatted at 2026-06-19 12:30:09 on dist-test-slave-fg3h"
server_key: "f61e85cfac4e97d14acb9e67e367a0e9"
server_key_iv: "f97296a25ad43b1067748a67eb2cc7e9"
server_key_version: "encryptionkey@0"
I20260619 12:30:09.331014 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:09.349565 3527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:09.352937 3527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:09.353170 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:09.353755 3527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:09.354557 3527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:09.354602 3527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:09.354662 3527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:09.354692 3527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:09.363734 3527 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34183
I20260619 12:30:09.363763 3655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34183 every 8 connection(s)
I20260619 12:30:09.364634 3527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:30:09.367208 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 3527
I20260619 12:30:09.367337 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:30:09.367645 392 external_mini_cluster.cc:1428] Setting key dc34afe58664bdfb60e1b44dc94d8ac3
Jun 19 12:30:09 dist-test-slave-fg3h krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:30:09.379112 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:09.366482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44247 (local address 127.0.98.62:42623)
0619 12:30:09.366610 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:09.366614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:09.367571 (+ 957us) server_negotiation.cc:407] Connection header received
0619 12:30:09.369200 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:09.369203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:09.369263 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:09.369350 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:09.370532 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:09.371053 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:09.372186 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:09.372420 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:09.372525 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:09.375206 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:09.375209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:09.375212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:09.375235 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:09.376835 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:09.377433 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:09.377437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:09.377439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:09.377550 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:09.377932 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:09.377935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:09.377936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:09.378113 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:09.378194 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:09.378832 (+ 638us) server_negotiation.cc:299] Negotiation successful
0619 12:30:09.378960 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 12:30:09.380069 3658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:09.366867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42623 (local address 127.0.98.1:44247)
0619 12:30:09.367430 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:09.367480 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:09.368934 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:09.369556 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:09.369563 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:09.369915 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:09.370392 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:09.370406 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:09.371178 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:09.371182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:09.372066 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:09.372075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:09.372574 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:09.372578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:09.372751 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:09.373430 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:09.373446 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:09.375014 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:09.376966 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:09.376972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:09.376988 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:09.377318 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:09.377675 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:09.377678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:09.377679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:09.377837 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:09.378229 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:09.378235 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:09.378575 (+ 340us) client_negotiation.cc:769] Sending connection context
0619 12:30:09.378735 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 12:30:09.379145 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":137,"threads_started":1}
I20260619 12:30:09.381538 3656 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42623
I20260619 12:30:09.381824 3656 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:09.382421 3656 heartbeater.cc:511] Master 127.0.98.62:42623 requested a full tablet report, sending...
I20260619 12:30:09.384274 3465 ts_manager.cc:194] Registered new tserver with Master: 57cf7df1fa6840198c18bf54b5b1de69 (127.0.98.1:34183)
I20260619 12:30:09.385957 3465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:30:09.395941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42623
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:09.506407 3661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:09.506659 3661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:09.506702 3661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:09.510686 3661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:09.510740 3661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:09.510799 3661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:30:09.516003 3661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:09.517025 3661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:09.518065 3661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:09.527814 3667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:09.527796 3669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:09.527796 3666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:09.527865 3661 server_base.cc:1034] running on GCE node
I20260619 12:30:09.528623 3661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:09.529320 3661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:09.530555 3661 hybrid_clock.cc:648] HybridClock initialized: now 1781872209530531 us; error 40 us; skew 500 ppm
Jun 19 12:30:09 dist-test-slave-fg3h krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:09.533416 3661 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:30:09.534405 3661 webserver.cc:460] Webserver started at http://127.0.98.2:45425/ using document root <none> and password file <none>
I20260619 12:30:09.534965 3661 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:09.535025 3661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:09.535240 3661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:09.537387 3661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "0e73f708b46a42f7bd73a1726e9fabf4"
format_stamp: "Formatted at 2026-06-19 12:30:09 on dist-test-slave-fg3h"
server_key: "868c4f5029fc2df6f5eb843a2592258c"
server_key_iv: "13b287e4f0c06b704a2d9c361a81328c"
server_key_version: "encryptionkey@0"
I20260619 12:30:09.537931 3661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0e73f708b46a42f7bd73a1726e9fabf4"
format_stamp: "Formatted at 2026-06-19 12:30:09 on dist-test-slave-fg3h"
server_key: "868c4f5029fc2df6f5eb843a2592258c"
server_key_iv: "13b287e4f0c06b704a2d9c361a81328c"
server_key_version: "encryptionkey@0"
I20260619 12:30:09.541626 3661 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 12:30:09.543978 3676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:09.545354 3661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:09.545498 3661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "0e73f708b46a42f7bd73a1726e9fabf4"
format_stamp: "Formatted at 2026-06-19 12:30:09 on dist-test-slave-fg3h"
server_key: "868c4f5029fc2df6f5eb843a2592258c"
server_key_iv: "13b287e4f0c06b704a2d9c361a81328c"
server_key_version: "encryptionkey@0"
I20260619 12:30:09.545603 3661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:09.574967 3661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:09.578164 3661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:09.578442 3661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:09.579048 3661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:09.579896 3661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:09.579963 3661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:09.580031 3661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:09.580076 3661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:09.589066 3661 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46285
I20260619 12:30:09.589109 3789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46285 every 8 connection(s)
I20260619 12:30:09.589962 3661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:30:09.592295 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 3661
I20260619 12:30:09.592427 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:30:09.592707 392 external_mini_cluster.cc:1428] Setting key aca6657a03d607dcdfc1ae100fb80fa6
Jun 19 12:30:09 dist-test-slave-fg3h krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:30:09.603943 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:09.591773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53623 (local address 127.0.98.62:42623)
0619 12:30:09.591894 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:09.591898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:09.592684 (+ 786us) server_negotiation.cc:407] Connection header received
0619 12:30:09.594314 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:09.594317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:09.594368 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:09.594444 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:09.595548 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:09.596068 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:09.597240 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:09.597475 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:09.597581 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:09.600074 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:09.600082 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:09.600084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:09.600106 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:09.601954 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:09.602465 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:09.602470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:09.602472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:09.602554 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:09.602969 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:09.602971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:09.602972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:09.603138 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:09.603240 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:09.603725 (+ 485us) server_negotiation.cc:299] Negotiation successful
0619 12:30:09.603823 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 12:30:09.604820 3792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:09.592101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42623 (local address 127.0.98.2:53623)
0619 12:30:09.592537 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:09.592587 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:09.594099 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:09.594603 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:09.594610 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:09.594954 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:09.595411 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:09.595424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:09.596187 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:09.596192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:09.597067 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:09.597076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:09.597616 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:09.597620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:09.597790 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:09.598439 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:09.598455 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:09.599917 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:09.602077 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:09.602082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:09.602096 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:09.602366 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:09.602639 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:09.602644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:09.602646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:09.602777 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:09.603239 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:09.603246 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:09.603483 (+ 237us) client_negotiation.cc:769] Sending connection context
0619 12:30:09.603678 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 12:30:09.603953 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":130,"threads_started":1}
I20260619 12:30:09.606212 3790 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42623
I20260619 12:30:09.606492 3790 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:09.607086 3790 heartbeater.cc:511] Master 127.0.98.62:42623 requested a full tablet report, sending...
I20260619 12:30:09.608031 3465 ts_manager.cc:194] Registered new tserver with Master: 0e73f708b46a42f7bd73a1726e9fabf4 (127.0.98.2:46285)
I20260619 12:30:09.608677 3465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:30:09.620172 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42623
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:09.730945 3795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:09.731240 3795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:09.731307 3795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:09.735394 3795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:09.735471 3795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:09.735554 3795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:30:09.740893 3795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:09.742024 3795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:09.743031 3795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:09.752516 3801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:09.752560 3803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:09.752594 3800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:09.752915 3795 server_base.cc:1034] running on GCE node
I20260619 12:30:09.753377 3795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:09.754007 3795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:09.755177 3795 hybrid_clock.cc:648] HybridClock initialized: now 1781872209755164 us; error 25 us; skew 500 ppm
Jun 19 12:30:09 dist-test-slave-fg3h krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:09.758484 3795 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:30:09.759371 3795 webserver.cc:460] Webserver started at http://127.0.98.3:42849/ using document root <none> and password file <none>
I20260619 12:30:09.759923 3795 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:09.759985 3795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:09.760201 3795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:09.762264 3795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "819736ad91a6434485355efe32c8b443"
format_stamp: "Formatted at 2026-06-19 12:30:09 on dist-test-slave-fg3h"
server_key: "31ae1d651a8cfebbb1a116713ae3eaff"
server_key_iv: "ebb602d29f859874285c1a0853bdc16e"
server_key_version: "encryptionkey@0"
I20260619 12:30:09.762825 3795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "819736ad91a6434485355efe32c8b443"
format_stamp: "Formatted at 2026-06-19 12:30:09 on dist-test-slave-fg3h"
server_key: "31ae1d651a8cfebbb1a116713ae3eaff"
server_key_iv: "ebb602d29f859874285c1a0853bdc16e"
server_key_version: "encryptionkey@0"
I20260619 12:30:09.766702 3795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:30:09.769157 3810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:09.770419 3795 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:30:09.770537 3795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "819736ad91a6434485355efe32c8b443"
format_stamp: "Formatted at 2026-06-19 12:30:09 on dist-test-slave-fg3h"
server_key: "31ae1d651a8cfebbb1a116713ae3eaff"
server_key_iv: "ebb602d29f859874285c1a0853bdc16e"
server_key_version: "encryptionkey@0"
I20260619 12:30:09.770639 3795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:09.789834 3795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:09.792886 3795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:09.793112 3795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:09.793702 3795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:09.794502 3795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:09.794548 3795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:09.794612 3795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:09.794649 3795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:09.803761 3795 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37573
I20260619 12:30:09.803833 3923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37573 every 8 connection(s)
I20260619 12:30:09.804656 3795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:30:09.806188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 3795
I20260619 12:30:09.806331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:30:09.806628 392 external_mini_cluster.cc:1428] Setting key 1b84374f30a6d4919b8b3c5b10c9c0d5
Jun 19 12:30:09 dist-test-slave-fg3h krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:09.818190 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:09.807005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44125 (local address 127.0.98.62:42623)
0619 12:30:09.807342 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:09.807347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:09.807553 (+ 206us) server_negotiation.cc:407] Connection header received
0619 12:30:09.809263 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:09.809266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:09.809312 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:09.809435 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:09.810615 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:09.811138 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:09.812322 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:09.812538 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:09.812628 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:09.815000 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:09.815003 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:09.815005 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:09.815026 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:09.816260 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:09.816752 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:09.816754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:09.816756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:09.816841 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:09.817203 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:09.817205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:09.817207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:09.817385 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:09.817464 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:09.817968 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 12:30:09.818072 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260619 12:30:09.819048 3926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:09.806871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42623 (local address 127.0.98.3:44125)
0619 12:30:09.807360 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:09.807452 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:09.809001 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:09.809588 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:09.809594 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:09.809941 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:09.810454 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:09.810468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:09.811246 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:09.811249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:09.812187 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:09.812196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:09.812672 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:09.812675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:09.812817 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:09.813443 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:09.813459 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:09.814851 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:09.816370 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:09.816409 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:09.816422 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:09.816668 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:09.816918 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:09.816920 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:09.816921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:09.817051 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:09.817480 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:09.817485 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:09.817723 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 12:30:09.817870 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 12:30:09.818119 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":124,"threads_started":1}
I20260619 12:30:09.820355 3924 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42623
I20260619 12:30:09.820605 3924 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:09.821187 3924 heartbeater.cc:511] Master 127.0.98.62:42623 requested a full tablet report, sending...
I20260619 12:30:09.822111 3465 ts_manager.cc:194] Registered new tserver with Master: 819736ad91a6434485355efe32c8b443 (127.0.98.3:37573)
I20260619 12:30:09.822674 3465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44125
I20260619 12:30:09.832572 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 12:30:09 dist-test-slave-fg3h krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872209, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 12:30:09 dist-test-slave-fg3h krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872209, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:09.860647 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:09.851578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34492 (local address 127.0.98.62:42623)
0619 12:30:09.851753 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:09.851758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:09.851903 (+ 145us) server_negotiation.cc:407] Connection header received
0619 12:30:09.852072 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:09.852075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:09.852117 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:09.852204 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:09.852941 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:09.853535 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:09.854376 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:09.854593 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:09.854691 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:09.857580 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:09.857583 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:09.857585 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:09.857612 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:09.859040 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:09.859474 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:09.859477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:09.859478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:09.859557 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:09.859797 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:09.859799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:09.859800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:09.860014 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:09.860116 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:09.860388 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 12:30:09.860517 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 12:30:09.861187 3465 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34492
I20260619 12:30:09.862300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 3527
I20260619 12:30:09.868384 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 3661
I20260619 12:30:09.874013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 3795
I20260619 12:30:09.879020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 3433
2026-06-19T12:30:09Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2143 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:30:09 dist-test-slave-fg3h krb5kdc[3936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:30:09 dist-test-slave-fg3h krb5kdc[3936](info): set up 2 sockets
Jun 19 12:30:09 dist-test-slave-fg3h krb5kdc[3936](info): commencing operation
krb5kdc: starting...
W20260619 12:30:10.937238 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:30:11 dist-test-slave-fg3h krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:30:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:30:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:30:11.065783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40321
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:11.176795 3949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:11.177052 3949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:11.177143 3949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:11.181213 3949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:11.181265 3949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:11.181285 3949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:11.181303 3949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:11.181346 3949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:30:11.186599 3949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:11.187613 3949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:11.188629 3949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:11.196991 3957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:11.197077 3949 server_base.cc:1034] running on GCE node
W20260619 12:30:11.196997 3954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:11.196997 3955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:11.197839 3949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:11.198894 3949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:11.200098 3949 hybrid_clock.cc:648] HybridClock initialized: now 1781872211200068 us; error 52 us; skew 500 ppm
Jun 19 12:30:11 dist-test-slave-fg3h krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:11.202762 3949 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:30:11.203610 3949 webserver.cc:460] Webserver started at http://127.0.98.62:41635/ using document root <none> and password file <none>
I20260619 12:30:11.204159 3949 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:11.204247 3949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:11.204475 3949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:11.206542 3949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "838fb36e459d46588a6887820311b468"
format_stamp: "Formatted at 2026-06-19 12:30:11 on dist-test-slave-fg3h"
server_key: "57b80b32ce99d8fc37b8b41bd42f6f62"
server_key_iv: "f2a8985e697f83ccf0111c1e2137bf27"
server_key_version: "encryptionkey@0"
I20260619 12:30:11.207304 3949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "838fb36e459d46588a6887820311b468"
format_stamp: "Formatted at 2026-06-19 12:30:11 on dist-test-slave-fg3h"
server_key: "57b80b32ce99d8fc37b8b41bd42f6f62"
server_key_iv: "f2a8985e697f83ccf0111c1e2137bf27"
server_key_version: "encryptionkey@0"
I20260619 12:30:11.211050 3949 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 12:30:11.213547 3964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:11.214807 3949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:11.214944 3949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "838fb36e459d46588a6887820311b468"
format_stamp: "Formatted at 2026-06-19 12:30:11 on dist-test-slave-fg3h"
server_key: "57b80b32ce99d8fc37b8b41bd42f6f62"
server_key_iv: "f2a8985e697f83ccf0111c1e2137bf27"
server_key_version: "encryptionkey@0"
I20260619 12:30:11.215060 3949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:11.231848 3949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:11.235205 3949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:11.235397 3949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:11.242228 3949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40321
I20260619 12:30:11.242252 4016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40321 every 8 connection(s)
I20260619 12:30:11.243147 3949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:30:11.246155 4017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:11.251711 4017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468: Bootstrap starting.
I20260619 12:30:11.251863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 3949
I20260619 12:30:11.251986 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:30:11.252282 392 external_mini_cluster.cc:1428] Setting key 7d922118e4b3f2d61d929e31fe054548
I20260619 12:30:11.254406 4017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:11.255137 4017 log.cc:824] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:11.257609 4017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468: No bootstrap required, opened a new log
Jun 19 12:30:11 dist-test-slave-fg3h krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:11.262583 4017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838fb36e459d46588a6887820311b468" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40321 } }
I20260619 12:30:11.262871 4017 raft_consensus.cc:374] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:11.262964 4017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 838fb36e459d46588a6887820311b468, State: Initialized, Role: FOLLOWER
I20260619 12:30:11.263543 4017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838fb36e459d46588a6887820311b468" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40321 } }
I20260619 12:30:11.263739 4017 raft_consensus.cc:388] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:30:11.263814 4017 raft_consensus.cc:482] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:30:11.263918 4017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:11.265286 4017 raft_consensus.cc:504] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838fb36e459d46588a6887820311b468" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40321 } }
I20260619 12:30:11.265688 4017 leader_election.cc:302] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 838fb36e459d46588a6887820311b468; no voters:
I20260619 12:30:11.266080 4017 leader_election.cc:288] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:30:11.266207 4022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:11.266436 4022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468 [term 1 LEADER]: Becoming Leader. State: Replica: 838fb36e459d46588a6887820311b468, State: Running, Role: LEADER
I20260619 12:30:11.266798 4022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838fb36e459d46588a6887820311b468" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40321 } }
I20260619 12:30:11.267563 4017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:11.268419 4024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 838fb36e459d46588a6887820311b468. Latest consensus state: current_term: 1 leader_uuid: "838fb36e459d46588a6887820311b468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838fb36e459d46588a6887820311b468" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40321 } } }
I20260619 12:30:11.268617 4024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:11.269193 4023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "838fb36e459d46588a6887820311b468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838fb36e459d46588a6887820311b468" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40321 } } }
I20260619 12:30:11.269291 4023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:11.270023 4031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:30:11.270068 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:11.253850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47554 (local address 127.0.98.62:40321)
0619 12:30:11.254292 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:11.254309 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:11.254344 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:30:11.255202 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:11.255266 (+ 64us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:11.255715 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:11.256089 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:11.257195 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:11.258282 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:11.259607 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:11.260079 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:11.260211 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:11.263323 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:11.263329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:11.263347 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:11.263379 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:11.265824 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:11.266394 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:11.266401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:11.266408 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:11.266651 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:11.267010 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:11.267013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:11.267014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:11.267521 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:11.267719 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:11.268520 (+ 801us) server_negotiation.cc:299] Negotiation successful
0619 12:30:11.268853 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":145,"threads_started":1}
I20260619 12:30:11.273497 4031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:30:11.278815 4031 catalog_manager.cc:1347] Generated new cluster ID: ee386a2b198746c9a9ab4d04920cd3f4
I20260619 12:30:11.278887 4031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:30:11.297068 4031 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:30:11.298399 4031 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:30:11.309060 4031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 838fb36e459d46588a6887820311b468: Generated new TSK 0
I20260619 12:30:11.309988 4031 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:30:11.343686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40321
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:11.456936 4043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:11.457226 4043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:11.457310 4043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:11.461526 4043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:11.461589 4043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:11.461649 4043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:30:11.466888 4043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:11.468024 4043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:11.469241 4043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:11.478842 4048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:11.479079 4043 server_base.cc:1034] running on GCE node
W20260619 12:30:11.478907 4051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:11.478837 4049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:11.479571 4043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:11.480156 4043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:11.481324 4043 hybrid_clock.cc:648] HybridClock initialized: now 1781872211481292 us; error 43 us; skew 500 ppm
Jun 19 12:30:11 dist-test-slave-fg3h krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:11.484067 4043 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:30:11.484874 4043 webserver.cc:460] Webserver started at http://127.0.98.1:39801/ using document root <none> and password file <none>
I20260619 12:30:11.485443 4043 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:11.485510 4043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:11.485674 4043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:11.487628 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "832f19ea6d894e7fb41b0c357d155063"
format_stamp: "Formatted at 2026-06-19 12:30:11 on dist-test-slave-fg3h"
server_key: "45aa0db7d42761e5056c35fe525d49cc"
server_key_iv: "40ef0e89fb645dc34449e1ca842e630b"
server_key_version: "encryptionkey@0"
I20260619 12:30:11.488127 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "832f19ea6d894e7fb41b0c357d155063"
format_stamp: "Formatted at 2026-06-19 12:30:11 on dist-test-slave-fg3h"
server_key: "45aa0db7d42761e5056c35fe525d49cc"
server_key_iv: "40ef0e89fb645dc34449e1ca842e630b"
server_key_version: "encryptionkey@0"
I20260619 12:30:11.491976 4043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:30:11.494735 4058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:11.496027 4043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:30:11.496172 4043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "832f19ea6d894e7fb41b0c357d155063"
format_stamp: "Formatted at 2026-06-19 12:30:11 on dist-test-slave-fg3h"
server_key: "45aa0db7d42761e5056c35fe525d49cc"
server_key_iv: "40ef0e89fb645dc34449e1ca842e630b"
server_key_version: "encryptionkey@0"
I20260619 12:30:11.496290 4043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:11.526721 4043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:11.530218 4043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:11.530426 4043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:11.531037 4043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:11.531881 4043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:11.531955 4043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:11.532024 4043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:11.532078 4043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:11.540998 4043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46839
I20260619 12:30:11.541028 4171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46839 every 8 connection(s)
I20260619 12:30:11.541922 4043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:30:11.550369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 4043
I20260619 12:30:11.550499 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:30:11.550866 392 external_mini_cluster.cc:1428] Setting key 6f80279dfe0d4bcf2f461fd4787763e6
Jun 19 12:30:11 dist-test-slave-fg3h krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:11.556402 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:11.543787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60081 (local address 127.0.98.62:40321)
0619 12:30:11.543949 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:11.543953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:11.544805 (+ 852us) server_negotiation.cc:407] Connection header received
0619 12:30:11.546234 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:11.546237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:11.546283 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:11.546362 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:11.547559 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:11.548112 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:11.549282 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:11.549520 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:11.549609 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:11.552364 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:11.552367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:11.552377 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:11.552406 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:11.554213 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:11.554724 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:11.554728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:11.554730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:11.554828 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:11.555238 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:11.555240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:11.555242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:11.555410 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:11.555487 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:11.556159 (+ 672us) server_negotiation.cc:299] Negotiation successful
0619 12:30:11.556264 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 12:30:11.557667 4174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:11.544160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40321 (local address 127.0.98.1:60081)
0619 12:30:11.544642 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:11.544686 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:11.545990 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:11.546528 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:11.546535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:11.546912 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:11.547407 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:11.547420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:11.548236 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:11.548239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:11.549167 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:11.549177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:11.549682 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:11.549686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:11.549836 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:11.550403 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:11.550427 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:11.552194 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:11.554348 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:11.554352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:11.554364 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:11.554616 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:11.554926 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:11.554928 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:11.554929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:11.555138 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:11.555518 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:11.555526 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:11.555900 (+ 374us) client_negotiation.cc:769] Sending connection context
0619 12:30:11.556118 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 12:30:11.556495 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":123,"threads_started":1}
I20260619 12:30:11.561240 4172 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40321
I20260619 12:30:11.561614 4172 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:30:11.562265 4172 heartbeater.cc:511] Master 127.0.98.62:40321 requested a full tablet report, sending...
I20260619 12:30:11.564498 3981 ts_manager.cc:194] Registered new tserver with Master: 832f19ea6d894e7fb41b0c357d155063 (127.0.98.1:46839)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:30:11.566221 3981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:30:11.584156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40321
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:11.696648 4177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:11.696908 4177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:11.696985 4177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:11.701229 4177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:11.701289 4177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:11.701376 4177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:30:11.706594 4177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:11.707643 4177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:11.708666 4177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:11.718218 4182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:11.718230 4183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:11.718268 4177 server_base.cc:1034] running on GCE node
W20260619 12:30:11.718214 4185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:11.718926 4177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:11.719550 4177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:11.720726 4177 hybrid_clock.cc:648] HybridClock initialized: now 1781872211720677 us; error 67 us; skew 500 ppm
Jun 19 12:30:11 dist-test-slave-fg3h krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:11.723593 4177 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:30:11.724452 4177 webserver.cc:460] Webserver started at http://127.0.98.2:41565/ using document root <none> and password file <none>
I20260619 12:30:11.724985 4177 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:11.725044 4177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:11.725281 4177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:11.727209 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "3bf8c2ec1ed54bfd81f64d6e29ac86dd"
format_stamp: "Formatted at 2026-06-19 12:30:11 on dist-test-slave-fg3h"
server_key: "9c15bc018301d388791704b9822f0bd8"
server_key_iv: "fd17185fd0899dcdffd755b7ea0ad357"
server_key_version: "encryptionkey@0"
I20260619 12:30:11.727711 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3bf8c2ec1ed54bfd81f64d6e29ac86dd"
format_stamp: "Formatted at 2026-06-19 12:30:11 on dist-test-slave-fg3h"
server_key: "9c15bc018301d388791704b9822f0bd8"
server_key_iv: "fd17185fd0899dcdffd755b7ea0ad357"
server_key_version: "encryptionkey@0"
I20260619 12:30:11.731524 4177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 12:30:11.733995 4192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:11.735213 4177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 12:30:11.735312 4177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "3bf8c2ec1ed54bfd81f64d6e29ac86dd"
format_stamp: "Formatted at 2026-06-19 12:30:11 on dist-test-slave-fg3h"
server_key: "9c15bc018301d388791704b9822f0bd8"
server_key_iv: "fd17185fd0899dcdffd755b7ea0ad357"
server_key_version: "encryptionkey@0"
I20260619 12:30:11.735386 4177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:11.749804 4177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:11.752789 4177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:11.752988 4177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:11.753575 4177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:11.754379 4177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:11.754457 4177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:11.754529 4177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:11.754573 4177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:11.765590 4177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33771
I20260619 12:30:11.765682 4305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33771 every 8 connection(s)
I20260619 12:30:11.766512 4177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:30:11.770936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 4177
I20260619 12:30:11.771045 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:30:11.771461 392 external_mini_cluster.cc:1428] Setting key b63f962ba92bf9a2533d2e93a80521f2
Jun 19 12:30:11 dist-test-slave-fg3h krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:11.781791 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:11.768317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53111 (local address 127.0.98.62:40321)
0619 12:30:11.768452 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:11.768456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:11.769619 (+ 1163us) server_negotiation.cc:407] Connection header received
0619 12:30:11.771293 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:11.771296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:11.771360 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:11.771445 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:11.772843 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:11.773582 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:11.774786 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:11.775003 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:11.775113 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:11.777938 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:11.777944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:11.777950 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:11.777982 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:11.779670 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:11.780180 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:11.780186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:11.780189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:11.780282 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:11.780617 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:11.780620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:11.780622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:11.780846 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:11.780965 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:11.781491 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 12:30:11.781625 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:30:11.782548 4308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:11.768933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40321 (local address 127.0.98.2:53111)
0619 12:30:11.769428 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:11.769501 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:11.771044 (+ 1543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:11.771666 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:11.771676 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:11.772090 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:11.772710 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:11.772729 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:11.773720 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:11.773723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:11.774636 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:11.774646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:11.775170 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:11.775174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:11.775389 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:11.776004 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:11.776029 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:11.777771 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:11.779808 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:11.779813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:11.779824 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:11.780074 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:11.780386 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:11.780388 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:11.780390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:11.780515 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:11.780950 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:11.780955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:11.781253 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 12:30:11.781410 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 12:30:11.781655 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":173,"threads_started":1}
I20260619 12:30:11.783732 4306 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40321
I20260619 12:30:11.783962 4306 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 12:30:11.784437 4306 heartbeater.cc:511] Master 127.0.98.62:40321 requested a full tablet report, sending...
I20260619 12:30:11.785688 3981 ts_manager.cc:194] Registered new tserver with Master: 3bf8c2ec1ed54bfd81f64d6e29ac86dd (127.0.98.2:33771)
I20260619 12:30:11.786422 3981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:30:11.800848 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40321
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:11.913714 4311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:11.913970 4311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:11.914012 4311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:11.918332 4311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:11.918387 4311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:11.918445 4311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:30:11.923684 4311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:11.924746 4311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:11.925783 4311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:11.935216 4316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:11.935256 4319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:11.935232 4317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:11.935581 4311 server_base.cc:1034] running on GCE node
I20260619 12:30:11.935983 4311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:11.936684 4311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:11.937862 4311 hybrid_clock.cc:648] HybridClock initialized: now 1781872211937831 us; error 51 us; skew 500 ppm
Jun 19 12:30:11 dist-test-slave-fg3h krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:11.940742 4311 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:30:11.941691 4311 webserver.cc:460] Webserver started at http://127.0.98.3:41613/ using document root <none> and password file <none>
I20260619 12:30:11.942262 4311 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:11.942354 4311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:11.942592 4311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:11.944662 4311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "33f3bc4d7e4349ec9dbace3b2ffea254"
format_stamp: "Formatted at 2026-06-19 12:30:11 on dist-test-slave-fg3h"
server_key: "7efc22dc570e65e86996e515a37c1e92"
server_key_iv: "f4c7e75de2fe95c994b90dce0fa6029d"
server_key_version: "encryptionkey@0"
I20260619 12:30:11.945286 4311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "33f3bc4d7e4349ec9dbace3b2ffea254"
format_stamp: "Formatted at 2026-06-19 12:30:11 on dist-test-slave-fg3h"
server_key: "7efc22dc570e65e86996e515a37c1e92"
server_key_iv: "f4c7e75de2fe95c994b90dce0fa6029d"
server_key_version: "encryptionkey@0"
I20260619 12:30:11.949152 4311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:30:11.951705 4326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:11.953003 4311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:30:11.953171 4311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "33f3bc4d7e4349ec9dbace3b2ffea254"
format_stamp: "Formatted at 2026-06-19 12:30:11 on dist-test-slave-fg3h"
server_key: "7efc22dc570e65e86996e515a37c1e92"
server_key_iv: "f4c7e75de2fe95c994b90dce0fa6029d"
server_key_version: "encryptionkey@0"
I20260619 12:30:11.953289 4311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:11.966547 4311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:11.970391 4311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:11.970548 4311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:11.971061 4311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:11.971894 4311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:11.971942 4311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:11.971982 4311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:11.971997 4311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:11.980638 4311 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39445
I20260619 12:30:11.980751 4439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39445 every 8 connection(s)
I20260619 12:30:11.981598 4311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:30:11.987468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 4311
I20260619 12:30:11.987600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:30:11.987913 392 external_mini_cluster.cc:1428] Setting key 54d608f67d244fc243bccf3f895634b8
Jun 19 12:30:11 dist-test-slave-fg3h krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:11.995241 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:11.983400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35757 (local address 127.0.98.62:40321)
0619 12:30:11.983530 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:11.983534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:11.984295 (+ 761us) server_negotiation.cc:407] Connection header received
0619 12:30:11.985789 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:11.985792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:11.985847 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:11.985951 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:11.987112 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:11.987742 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:11.989121 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:11.989352 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:11.989464 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:11.991861 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:11.991865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:11.991868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:11.991902 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:11.993288 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:11.993788 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:11.993792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:11.993794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:11.993877 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:11.994229 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:11.994234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:11.994235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:11.994461 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:11.994560 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:11.995029 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 12:30:11.995124 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:30:11.996126 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:11.983706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40321 (local address 127.0.98.3:35757)
0619 12:30:11.984143 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:11.984184 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:11.985580 (+ 1396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:11.986131 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:11.986139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:11.986485 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:11.986966 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:11.986981 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:11.987872 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:11.987876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:11.988989 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:11.988999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:11.989500 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:11.989504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:11.989680 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:11.990223 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:11.990239 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:11.991688 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:11.993403 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:11.993409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:11.993425 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:11.993677 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:11.993969 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:11.993974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:11.993976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:11.994114 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:11.994575 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:11.994581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:11.994817 (+ 236us) client_negotiation.cc:769] Sending connection context
0619 12:30:11.994990 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 12:30:11.995270 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":110,"threads_started":1}
I20260619 12:30:11.997319 4440 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40321
I20260619 12:30:11.997550 4440 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:11.998020 4440 heartbeater.cc:511] Master 127.0.98.62:40321 requested a full tablet report, sending...
I20260619 12:30:11.999097 3981 ts_manager.cc:194] Registered new tserver with Master: 33f3bc4d7e4349ec9dbace3b2ffea254 (127.0.98.3:39445)
I20260619 12:30:11.999768 3981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35757
I20260619 12:30:12.001945 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 12:30:12 dist-test-slave-fg3h krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 12:30:12 dist-test-slave-fg3h krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:30:12.028604 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:12.018577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40138 (local address 127.0.98.1:46839)
0619 12:30:12.018920 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:12.018938 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:12.018966 (+ 28us) server_negotiation.cc:407] Connection header received
0619 12:30:12.019146 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:12.019153 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:12.019356 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:12.019505 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:12.020276 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:12.021059 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:12.021968 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:12.022376 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:12.022508 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:12.025035 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:12.025044 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:12.025057 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:12.025125 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:12.026752 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:12.027203 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:12.027213 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:12.027215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:12.027293 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:12.027570 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:12.027574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:12.027576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:12.027838 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:12.028086 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:12.028253 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 12:30:12.028412 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":116,"threads_started":1}
Jun 19 12:30:12 dist-test-slave-fg3h krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:12.040217 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:12.031713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47570 (local address 127.0.98.62:40321)
0619 12:30:12.031873 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:12.031876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:12.031891 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:30:12.032015 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:12.032018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:12.032064 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:12.032137 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:12.032781 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:12.033370 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:12.034169 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:12.034380 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:12.034463 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:12.036926 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:12.036929 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:12.036931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:12.036954 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:12.038614 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:12.039030 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:12.039032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:12.039033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:12.039104 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:12.039388 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:12.039390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:12.039391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:12.039599 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:12.039736 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:12.039926 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 12:30:12.040017 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 12:30:12.040616 3981 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47570
I20260619 12:30:12.041361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 4043
I20260619 12:30:12.047093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 4177
I20260619 12:30:12.053442 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 4311
I20260619 12:30:12.059249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 3949
2026-06-19T12:30:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2180 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:30:12 dist-test-slave-fg3h krb5kdc[4452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:30:12 dist-test-slave-fg3h krb5kdc[4452](info): set up 2 sockets
Jun 19 12:30:12 dist-test-slave-fg3h krb5kdc[4452](info): commencing operation
krb5kdc: starting...
W20260619 12:30:13.109398 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:30:13 dist-test-slave-fg3h krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:30:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:30:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:30:13.232530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38847
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:13.341799 4465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:13.342058 4465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:13.342100 4465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:13.346154 4465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:13.346213 4465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:13.346233 4465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:13.346251 4465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:13.346316 4465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:30:13.351569 4465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:13.352596 4465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:13.353627 4465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:13.361706 4471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:13.361714 4470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:13.361706 4473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:13.362061 4465 server_base.cc:1034] running on GCE node
I20260619 12:30:13.362613 4465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:13.363647 4465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:13.364849 4465 hybrid_clock.cc:648] HybridClock initialized: now 1781872213364822 us; error 52 us; skew 500 ppm
Jun 19 12:30:13 dist-test-slave-fg3h krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:13.372510 4465 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:30:13.373373 4465 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:13.373468 4465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:13.373700 4465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:13.375839 4465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "841f409f662e42c79fdcee030835b912"
format_stamp: "Formatted at 2026-06-19 12:30:13 on dist-test-slave-fg3h"
server_key: "f4ff3d9ea1578ae11b58c428104e1510"
server_key_iv: "e9284dda8733d613e5532450fc1178c8"
server_key_version: "encryptionkey@0"
I20260619 12:30:13.376416 4465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "841f409f662e42c79fdcee030835b912"
format_stamp: "Formatted at 2026-06-19 12:30:13 on dist-test-slave-fg3h"
server_key: "f4ff3d9ea1578ae11b58c428104e1510"
server_key_iv: "e9284dda8733d613e5532450fc1178c8"
server_key_version: "encryptionkey@0"
I20260619 12:30:13.380182 4465 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 12:30:13.382658 4479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:13.383942 4465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:30:13.384090 4465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "841f409f662e42c79fdcee030835b912"
format_stamp: "Formatted at 2026-06-19 12:30:13 on dist-test-slave-fg3h"
server_key: "f4ff3d9ea1578ae11b58c428104e1510"
server_key_iv: "e9284dda8733d613e5532450fc1178c8"
server_key_version: "encryptionkey@0"
I20260619 12:30:13.384202 4465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:13.395593 4465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:13.398536 4465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:13.398731 4465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:13.405583 4465 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38847
I20260619 12:30:13.405858 4531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38847 every 8 connection(s)
I20260619 12:30:13.406273 4465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:30:13.408855 4532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:13.409229 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 4465
I20260619 12:30:13.409360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:30:13.409643 392 external_mini_cluster.cc:1428] Setting key ded517b48b7da0cb3172ee023a643f3a
I20260619 12:30:13.415300 4532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912: Bootstrap starting.
I20260619 12:30:13.417917 4532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912: Neither blocks nor log segments found. Creating new log.
Jun 19 12:30:13 dist-test-slave-fg3h krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:13.418752 4532 log.cc:824] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:13.420743 4532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912: No bootstrap required, opened a new log
I20260619 12:30:13.423657 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:13.411032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48218 (local address 127.0.98.62:38847)
0619 12:30:13.411456 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:13.411470 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:13.411503 (+ 33us) server_negotiation.cc:407] Connection header received
0619 12:30:13.412199 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:13.412215 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:13.412523 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:13.412863 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:13.413655 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:13.414602 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:13.415487 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:13.416144 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:13.416332 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:13.419017 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:13.419025 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:13.419045 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:13.419079 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:13.420885 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:13.421405 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:13.421410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:13.421414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:13.421522 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:13.421829 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:13.421831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:13.421832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:13.422200 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:13.422343 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:13.422635 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 12:30:13.422972 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":142,"threads_started":1}
I20260619 12:30:13.424822 4532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841f409f662e42c79fdcee030835b912" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38847 } }
I20260619 12:30:13.425033 4532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:13.425155 4532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841f409f662e42c79fdcee030835b912, State: Initialized, Role: FOLLOWER
I20260619 12:30:13.425601 4532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841f409f662e42c79fdcee030835b912" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38847 } }
I20260619 12:30:13.425769 4532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:30:13.425838 4532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:30:13.425951 4532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:13.426856 4532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841f409f662e42c79fdcee030835b912" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38847 } }
I20260619 12:30:13.427187 4532 leader_election.cc:302] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 841f409f662e42c79fdcee030835b912; no voters:
I20260619 12:30:13.427476 4532 leader_election.cc:288] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:30:13.427618 4537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:13.427930 4537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912 [term 1 LEADER]: Becoming Leader. State: Replica: 841f409f662e42c79fdcee030835b912, State: Running, Role: LEADER
I20260619 12:30:13.428243 4537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841f409f662e42c79fdcee030835b912" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38847 } }
I20260619 12:30:13.428736 4532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:13.429844 4539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 841f409f662e42c79fdcee030835b912. Latest consensus state: current_term: 1 leader_uuid: "841f409f662e42c79fdcee030835b912" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841f409f662e42c79fdcee030835b912" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38847 } } }
I20260619 12:30:13.429975 4539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:13.430405 4546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:30:13.430390 4538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "841f409f662e42c79fdcee030835b912" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "841f409f662e42c79fdcee030835b912" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38847 } } }
I20260619 12:30:13.431356 4538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:13.433198 4546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:30:13.438364 4546 catalog_manager.cc:1347] Generated new cluster ID: 553ef876d3cc44cab90462a54fbbb834
I20260619 12:30:13.438426 4546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:30:13.446743 4546 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:30:13.447625 4546 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:30:13.454594 4546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912: Generated new TSK 0
I20260619 12:30:13.455261 4546 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:30:13.486013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38847
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:13.605140 4558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:13.605440 4558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:13.605490 4558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:13.610229 4558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:13.610304 4558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:13.610420 4558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:30:13.616504 4558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:13.617720 4558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:13.619098 4558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:13.628955 4564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:13.628979 4563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:13.629186 4558 server_base.cc:1034] running on GCE node
W20260619 12:30:13.628898 4566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:13.629725 4558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:13.630406 4558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:13.631569 4558 hybrid_clock.cc:648] HybridClock initialized: now 1781872213631551 us; error 30 us; skew 500 ppm
Jun 19 12:30:13 dist-test-slave-fg3h krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:13.634495 4558 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:30:13.635231 4558 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:13.635327 4558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:13.635568 4558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:13.637669 4558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "4ba7985b233a4df2b3819acd024bda3e"
format_stamp: "Formatted at 2026-06-19 12:30:13 on dist-test-slave-fg3h"
server_key: "8627c702fbe2990ccb3143f2fcace65c"
server_key_iv: "d8f4ab726f39d2f242a4c1c001e56826"
server_key_version: "encryptionkey@0"
I20260619 12:30:13.638265 4558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4ba7985b233a4df2b3819acd024bda3e"
format_stamp: "Formatted at 2026-06-19 12:30:13 on dist-test-slave-fg3h"
server_key: "8627c702fbe2990ccb3143f2fcace65c"
server_key_iv: "d8f4ab726f39d2f242a4c1c001e56826"
server_key_version: "encryptionkey@0"
I20260619 12:30:13.642012 4558 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:30:13.644464 4572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:13.645885 4558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:13.646039 4558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "4ba7985b233a4df2b3819acd024bda3e"
format_stamp: "Formatted at 2026-06-19 12:30:13 on dist-test-slave-fg3h"
server_key: "8627c702fbe2990ccb3143f2fcace65c"
server_key_iv: "d8f4ab726f39d2f242a4c1c001e56826"
server_key_version: "encryptionkey@0"
I20260619 12:30:13.646166 4558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:13.670940 4558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:13.674458 4558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:13.674669 4558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:13.675266 4558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:13.676122 4558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:13.676191 4558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:13.676271 4558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:13.676314 4558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:13.685757 4558 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36721
I20260619 12:30:13.685784 4685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36721 every 8 connection(s)
I20260619 12:30:13.686389 4558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:30:13.692847 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 4558
I20260619 12:30:13.693024 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:30:13.693328 392 external_mini_cluster.cc:1428] Setting key ac0ded28d1c8b326e11b69d8d686cc76
Jun 19 12:30:13 dist-test-slave-fg3h krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:13.700596 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:13.688334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53481 (local address 127.0.98.62:38847)
0619 12:30:13.688479 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:13.688485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:13.689425 (+ 940us) server_negotiation.cc:407] Connection header received
0619 12:30:13.690777 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:13.690780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:13.690833 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:13.690909 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:13.692092 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:13.692601 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:13.693949 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:13.694213 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:13.694299 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:13.696872 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:13.696875 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:13.696878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:13.696900 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:13.698532 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:13.699063 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:13.699067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:13.699069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:13.699156 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:13.699504 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:13.699508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:13.699510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:13.699696 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:13.699804 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:13.700300 (+ 496us) server_negotiation.cc:299] Negotiation successful
0619 12:30:13.700432 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 12:30:13.701478 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:13.688768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38847 (local address 127.0.98.1:53481)
0619 12:30:13.689271 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:13.689314 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:13.690571 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:13.691101 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:13.691107 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:13.691465 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:13.691958 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:13.691971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:13.692738 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:13.692741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:13.693840 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:13.693850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:13.694385 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:13.694389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:13.694572 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:13.695230 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:13.695246 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:13.696689 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:13.698654 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:13.698660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:13.698675 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:13.698953 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:13.699257 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:13.699260 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:13.699262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:13.699408 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:13.699803 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:13.699811 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:13.700069 (+ 258us) client_negotiation.cc:769] Sending connection context
0619 12:30:13.700261 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 12:30:13.700584 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:30:13.702817 4686 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38847
I20260619 12:30:13.703087 4686 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:13.703552 4686 heartbeater.cc:511] Master 127.0.98.62:38847 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:30:13.705562 4496 ts_manager.cc:194] Registered new tserver with Master: 4ba7985b233a4df2b3819acd024bda3e (127.0.98.1:36721)
I20260619 12:30:13.707103 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:30:13.720909 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38847
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:13.847573 4691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:13.847898 4691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:13.847978 4691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:13.852432 4691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:13.852532 4691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:13.852632 4691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:30:13.858289 4691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:13.859431 4691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:13.860668 4691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:13.870572 4696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:13.870621 4691 server_base.cc:1034] running on GCE node
W20260619 12:30:13.870616 4697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:13.870616 4699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:13.871248 4691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:13.871915 4691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:13.873157 4691 hybrid_clock.cc:648] HybridClock initialized: now 1781872213873136 us; error 28 us; skew 500 ppm
Jun 19 12:30:13 dist-test-slave-fg3h krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:13.876057 4691 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:30:13.876865 4691 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:13.876960 4691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:13.877251 4691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:13.879351 4691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "ad34e831164d4a259d12bb31a9c41136"
format_stamp: "Formatted at 2026-06-19 12:30:13 on dist-test-slave-fg3h"
server_key: "d34b54187c39ee85b62df9dec262a568"
server_key_iv: "271a96d8fd8fb08bad6f05906c9a568a"
server_key_version: "encryptionkey@0"
I20260619 12:30:13.879958 4691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ad34e831164d4a259d12bb31a9c41136"
format_stamp: "Formatted at 2026-06-19 12:30:13 on dist-test-slave-fg3h"
server_key: "d34b54187c39ee85b62df9dec262a568"
server_key_iv: "271a96d8fd8fb08bad6f05906c9a568a"
server_key_version: "encryptionkey@0"
I20260619 12:30:13.883737 4691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 12:30:13.886271 4705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:13.887514 4691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:30:13.887677 4691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "ad34e831164d4a259d12bb31a9c41136"
format_stamp: "Formatted at 2026-06-19 12:30:13 on dist-test-slave-fg3h"
server_key: "d34b54187c39ee85b62df9dec262a568"
server_key_iv: "271a96d8fd8fb08bad6f05906c9a568a"
server_key_version: "encryptionkey@0"
I20260619 12:30:13.887802 4691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:13.924031 4691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:13.927098 4691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:13.927414 4691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:13.928020 4691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:13.928931 4691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:13.929008 4691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:13.929076 4691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:13.929152 4691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 12:30:13.938023 4691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38505
I20260619 12:30:13.938040 4818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38505 every 8 connection(s)
I20260619 12:30:13.938616 4691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 12:30:13 dist-test-slave-fg3h krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:13.947845 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 4691
I20260619 12:30:13.947974 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:30:13.948257 392 external_mini_cluster.cc:1428] Setting key f9617e325613c4af9c07d3f4e8488f42
I20260619 12:30:13.952299 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:13.940247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34793 (local address 127.0.98.62:38847)
0619 12:30:13.940378 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:13.940382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:13.941303 (+ 921us) server_negotiation.cc:407] Connection header received
0619 12:30:13.942643 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:13.942646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:13.942693 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:13.942769 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:13.943935 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:13.944445 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:13.945572 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:13.945819 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:13.945908 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:13.948490 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:13.948495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:13.948497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:13.948524 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:13.950263 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:13.950769 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:13.950772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:13.950774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:13.950870 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:13.951248 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:13.951253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:13.951254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:13.951441 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:13.951557 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:13.952032 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 12:30:13.952148 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 12:30:13.953183 4821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:13.940641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38847 (local address 127.0.98.2:34793)
0619 12:30:13.941158 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:13.941201 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:13.942442 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:13.942930 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:13.942937 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:13.943329 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:13.943810 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:13.943824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:13.944548 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:13.944551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:13.945460 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:13.945469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:13.945953 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:13.945955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:13.946115 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:13.946737 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:13.946754 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:13.948314 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:13.950377 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:13.950382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:13.950395 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:13.950650 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:13.951007 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:13.951009 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:13.951010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:13.951146 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:13.951554 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:13.951559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:13.951798 (+ 239us) client_negotiation.cc:769] Sending connection context
0619 12:30:13.951996 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 12:30:13.952314 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":103,"threads_started":1}
I20260619 12:30:13.954377 4819 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38847
I20260619 12:30:13.954641 4819 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:13.955111 4819 heartbeater.cc:511] Master 127.0.98.62:38847 requested a full tablet report, sending...
I20260619 12:30:13.956153 4496 ts_manager.cc:194] Registered new tserver with Master: ad34e831164d4a259d12bb31a9c41136 (127.0.98.2:38505)
I20260619 12:30:13.956735 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34793
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:30:13.975564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38847
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:14.092157 4824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:14.092568 4824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:14.092649 4824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:14.097939 4824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:14.098016 4824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:14.098081 4824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:30:14.103641 4824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:14.104794 4824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:14.105922 4824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:14.115877 4830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:14.115877 4829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:14.115958 4824 server_base.cc:1034] running on GCE node
W20260619 12:30:14.115919 4832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:14.116614 4824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:14.117375 4824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:14.118552 4824 hybrid_clock.cc:648] HybridClock initialized: now 1781872214118513 us; error 51 us; skew 500 ppm
Jun 19 12:30:14 dist-test-slave-fg3h krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:14.121469 4824 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:30:14.122154 4824 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:14.122217 4824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:14.122380 4824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:14.124375 4824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "deda46a8042c442cb0229ee0444f61da"
format_stamp: "Formatted at 2026-06-19 12:30:14 on dist-test-slave-fg3h"
server_key: "8d1497900b9f9021ce939a7342a6017e"
server_key_iv: "4376c6d11eb8dcff2e2f5cc4e33918e4"
server_key_version: "encryptionkey@0"
I20260619 12:30:14.124914 4824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "deda46a8042c442cb0229ee0444f61da"
format_stamp: "Formatted at 2026-06-19 12:30:14 on dist-test-slave-fg3h"
server_key: "8d1497900b9f9021ce939a7342a6017e"
server_key_iv: "4376c6d11eb8dcff2e2f5cc4e33918e4"
server_key_version: "encryptionkey@0"
I20260619 12:30:14.128666 4824 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 12:30:14.131335 4838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:14.132645 4824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 12:30:14.132766 4824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "deda46a8042c442cb0229ee0444f61da"
format_stamp: "Formatted at 2026-06-19 12:30:14 on dist-test-slave-fg3h"
server_key: "8d1497900b9f9021ce939a7342a6017e"
server_key_iv: "4376c6d11eb8dcff2e2f5cc4e33918e4"
server_key_version: "encryptionkey@0"
I20260619 12:30:14.132879 4824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:14.147643 4824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:14.151134 4824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:14.151350 4824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:14.151976 4824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:14.152825 4824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:14.152874 4824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:14.152915 4824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:14.152930 4824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:14.162247 4824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32903
I20260619 12:30:14.162340 4951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32903 every 8 connection(s)
I20260619 12:30:14.162875 4824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:30:14.167299 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 4824
I20260619 12:30:14.167405 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:30:14.167629 392 external_mini_cluster.cc:1428] Setting key a73ebdba21b5ba0be4b9b059688c2b54
Jun 19 12:30:14 dist-test-slave-fg3h krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:14.177485 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.164653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51649 (local address 127.0.98.62:38847)
0619 12:30:14.164811 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:14.164816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:14.165770 (+ 954us) server_negotiation.cc:407] Connection header received
0619 12:30:14.167594 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:14.167597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:14.167647 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:14.167766 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:14.169282 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.169935 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.171086 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.171298 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.171392 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:14.173977 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:14.173980 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:14.173982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:14.174008 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:14.175595 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:14.176084 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:14.176086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:14.176088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:14.176160 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:14.176450 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:14.176452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:14.176453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:14.176627 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:14.176698 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:14.177252 (+ 554us) server_negotiation.cc:299] Negotiation successful
0619 12:30:14.177357 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:30:14.178442 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.165169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38847 (local address 127.0.98.3:51649)
0619 12:30:14.165612 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:14.165656 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:14.166940 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:14.168006 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:14.168014 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:14.168440 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:14.169127 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:14.169151 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.170051 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.170054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:14.170964 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:14.170973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.171432 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.171435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:14.171586 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:14.172206 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:14.172222 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:14.173783 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:14.175710 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:14.175715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:14.175728 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:14.175988 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:14.176236 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:14.176238 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:14.176239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:14.176369 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:14.176709 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:14.176714 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:14.176958 (+ 244us) client_negotiation.cc:769] Sending connection context
0619 12:30:14.177208 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 12:30:14.177496 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
I20260619 12:30:14.179611 4952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38847
I20260619 12:30:14.179826 4952 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:14.180361 4952 heartbeater.cc:511] Master 127.0.98.62:38847 requested a full tablet report, sending...
I20260619 12:30:14.181556 4493 ts_manager.cc:194] Registered new tserver with Master: deda46a8042c442cb0229ee0444f61da (127.0.98.3:32903)
I20260619 12:30:14.182163 4493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51649
I20260619 12:30:14.194139 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:30:14.203955 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.196430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48234 (local address 127.0.98.62:38847)
0619 12:30:14.196592 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:14.196596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:14.196738 (+ 142us) server_negotiation.cc:407] Connection header received
0619 12:30:14.196830 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:14.196832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:14.196874 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:14.196942 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:14.197783 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.198308 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.199152 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.199364 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.199447 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:14.200755 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:14.200758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:14.200760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:14.200783 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:14.202337 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:14.202758 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:14.202761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:14.202762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:14.202843 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:14.203097 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:14.203099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:14.203100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:14.203306 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:14.203414 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:14.203611 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 12:30:14.203703 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 12:30:14.207170 4493 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:30:14.210491 4493 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:30:14.225024 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.219100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54680 (local address 127.0.98.2:38505)
0619 12:30:14.219811 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:14.219823 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:14.219847 (+ 24us) server_negotiation.cc:407] Connection header received
0619 12:30:14.219921 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:14.219929 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:14.220159 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:14.220694 (+ 535us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:14.220816 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.221730 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.223328 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.224256 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.224388 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:14.224574 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:14.224737 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 12:30:14.224858 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":160,"threads_started":1}
I20260619 12:30:14.225018 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.218901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38505 (local address 127.0.0.1:54680)
0619 12:30:14.219292 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:14.219345 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:14.219492 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:14.220342 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:14.220350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:14.220378 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:14.220567 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:14.220573 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.221857 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.221863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:14.223197 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:14.223206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.224395 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.224399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:14.224510 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:14.224548 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 12:30:14.224695 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 12:30:14.224843 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":60,"thread_start_us":89,"threads_started":1}
I20260619 12:30:14.225495 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.219672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32903 (local address 127.0.0.1:42226)
0619 12:30:14.220335 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:14.220348 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:14.220429 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:14.221397 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:14.221402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:14.221441 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:14.221537 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:14.221543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.223034 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.223037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:14.223868 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:14.223876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.225045 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.225049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:14.225188 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:14.225212 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 12:30:14.225278 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 12:30:14.225348 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":1,"spinlock_wait_cycles":5760,"thread_start_us":76,"threads_started":1}
I20260619 12:30:14.225912 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.220081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42226 (local address 127.0.98.3:32903)
0619 12:30:14.220623 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:14.220644 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:14.220667 (+ 23us) server_negotiation.cc:407] Connection header received
0619 12:30:14.220750 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:14.220756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:14.221012 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:14.221223 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:14.221681 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.222891 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.223995 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.224716 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.224837 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:14.224977 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:14.225664 (+ 687us) server_negotiation.cc:299] Negotiation successful
0619 12:30:14.225764 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":150,"threads_started":1}
I20260619 12:30:14.226414 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.219388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36721 (local address 127.0.0.1:47918)
0619 12:30:14.220312 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:14.220326 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:14.220438 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:14.221145 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:14.221148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:14.221166 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:14.221260 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:14.221266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.222748 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.222752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:14.224138 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:14.224149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.225389 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.225392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:14.226097 (+ 705us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:14.226122 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 12:30:14.226194 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 12:30:14.226259 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":56,"spinlock_wait_cycles":768,"thread_start_us":117,"threads_started":1}
I20260619 12:30:14.226951 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.219805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47918 (local address 127.0.98.1:36721)
0619 12:30:14.220179 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:14.220186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:14.220667 (+ 481us) server_negotiation.cc:407] Connection header received
0619 12:30:14.220750 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:14.220756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:14.220962 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:14.221176 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:14.221367 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.222604 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.224280 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.225078 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.225197 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:14.225332 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:14.226691 (+ 1359us) server_negotiation.cc:299] Negotiation successful
0619 12:30:14.226807 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":89,"threads_started":1}
I20260619 12:30:14.228519 4753 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff6fbd952ba45f7b233ddedbe97d846 (DEFAULT_TABLE table=test-table [id=b0e11ebe77904850929bf3a20d8fdb80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:14.228909 4886 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff6fbd952ba45f7b233ddedbe97d846 (DEFAULT_TABLE table=test-table [id=b0e11ebe77904850929bf3a20d8fdb80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:14.229537 4620 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff6fbd952ba45f7b233ddedbe97d846 (DEFAULT_TABLE table=test-table [id=b0e11ebe77904850929bf3a20d8fdb80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:14.230080 4886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff6fbd952ba45f7b233ddedbe97d846. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:14.230314 4753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff6fbd952ba45f7b233ddedbe97d846. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:14.230412 4620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff6fbd952ba45f7b233ddedbe97d846. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:14.236986 4969 tablet_bootstrap.cc:492] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da: Bootstrap starting.
I20260619 12:30:14.238040 4971 tablet_bootstrap.cc:492] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136: Bootstrap starting.
I20260619 12:30:14.238207 4970 tablet_bootstrap.cc:492] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e: Bootstrap starting.
I20260619 12:30:14.239975 4969 tablet_bootstrap.cc:654] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:14.240147 4970 tablet_bootstrap.cc:654] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:14.240146 4971 tablet_bootstrap.cc:654] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:14.241077 4970 log.cc:824] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:14.241258 4971 log.cc:824] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:14.241367 4969 log.cc:824] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:14.243001 4971 tablet_bootstrap.cc:492] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136: No bootstrap required, opened a new log
I20260619 12:30:14.243013 4970 tablet_bootstrap.cc:492] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e: No bootstrap required, opened a new log
I20260619 12:30:14.243152 4971 ts_tablet_manager.cc:1397] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:30:14.243217 4970 ts_tablet_manager.cc:1397] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 12:30:14.246049 4971 raft_consensus.cc:348] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad34e831164d4a259d12bb31a9c41136" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "deda46a8042c442cb0229ee0444f61da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32903 } } peers { permanent_uuid: "4ba7985b233a4df2b3819acd024bda3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } }
I20260619 12:30:14.246275 4971 raft_consensus.cc:374] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:14.246315 4971 raft_consensus.cc:729] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad34e831164d4a259d12bb31a9c41136, State: Initialized, Role: FOLLOWER
I20260619 12:30:14.246724 4971 consensus_queue.cc:260] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad34e831164d4a259d12bb31a9c41136" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "deda46a8042c442cb0229ee0444f61da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32903 } } peers { permanent_uuid: "4ba7985b233a4df2b3819acd024bda3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } }
I20260619 12:30:14.247342 4819 heartbeater.cc:503] Master 127.0.98.62:38847 was elected leader, sending a full tablet report...
I20260619 12:30:14.247310 4970 raft_consensus.cc:348] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad34e831164d4a259d12bb31a9c41136" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "deda46a8042c442cb0229ee0444f61da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32903 } } peers { permanent_uuid: "4ba7985b233a4df2b3819acd024bda3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } }
I20260619 12:30:14.247598 4970 raft_consensus.cc:374] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:14.247671 4970 raft_consensus.cc:729] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ba7985b233a4df2b3819acd024bda3e, State: Initialized, Role: FOLLOWER
I20260619 12:30:14.247696 4971 ts_tablet_manager.cc:1428] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 12:30:14.248112 4969 tablet_bootstrap.cc:492] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da: No bootstrap required, opened a new log
I20260619 12:30:14.248112 4970 consensus_queue.cc:260] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad34e831164d4a259d12bb31a9c41136" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "deda46a8042c442cb0229ee0444f61da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32903 } } peers { permanent_uuid: "4ba7985b233a4df2b3819acd024bda3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } }
I20260619 12:30:14.248448 4969 ts_tablet_manager.cc:1397] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20260619 12:30:14.248802 4686 heartbeater.cc:503] Master 127.0.98.62:38847 was elected leader, sending a full tablet report...
I20260619 12:30:14.249202 4970 ts_tablet_manager.cc:1428] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 12:30:14.252904 4969 raft_consensus.cc:348] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad34e831164d4a259d12bb31a9c41136" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "deda46a8042c442cb0229ee0444f61da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32903 } } peers { permanent_uuid: "4ba7985b233a4df2b3819acd024bda3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } }
I20260619 12:30:14.253203 4969 raft_consensus.cc:374] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:14.253296 4969 raft_consensus.cc:729] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deda46a8042c442cb0229ee0444f61da, State: Initialized, Role: FOLLOWER
I20260619 12:30:14.253755 4969 consensus_queue.cc:260] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad34e831164d4a259d12bb31a9c41136" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "deda46a8042c442cb0229ee0444f61da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32903 } } peers { permanent_uuid: "4ba7985b233a4df2b3819acd024bda3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } }
I20260619 12:30:14.254441 4952 heartbeater.cc:503] Master 127.0.98.62:38847 was elected leader, sending a full tablet report...
I20260619 12:30:14.254755 4969 ts_tablet_manager.cc:1428] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:30:14.289911 4976 raft_consensus.cc:482] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:30:14.290123 4976 raft_consensus.cc:504] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad34e831164d4a259d12bb31a9c41136" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "deda46a8042c442cb0229ee0444f61da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32903 } } peers { permanent_uuid: "4ba7985b233a4df2b3819acd024bda3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } }
I20260619 12:30:14.291464 4976 leader_election.cc:288] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad34e831164d4a259d12bb31a9c41136 (127.0.98.2:38505), deda46a8042c442cb0229ee0444f61da (127.0.98.3:32903)
I20260619 12:30:14.295873 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.291733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38505 (local address 127.0.98.1:53143)
0619 12:30:14.292077 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:14.292092 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:14.292232 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:14.292647 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:14.292650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:14.292700 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:14.292814 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:14.292820 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.293640 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.293646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:14.294692 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:14.294703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.295443 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.295446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:14.295552 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:14.295597 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 12:30:14.295674 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 12:30:14.295735 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":3,"spinlock_wait_cycles":8704,"thread_start_us":186,"threads_started":1}
I20260619 12:30:14.295938 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.291809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53143 (local address 127.0.98.2:38505)
0619 12:30:14.291935 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:14.291942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:14.292349 (+ 407us) server_negotiation.cc:407] Connection header received
0619 12:30:14.292421 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:14.292425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:14.292488 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:14.292578 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:14.292945 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.293527 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.294804 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.295320 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.295435 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:14.295495 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:14.295720 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 12:30:14.295793 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 12:30:14.296643 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.292406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36373 (local address 127.0.98.3:32903)
0619 12:30:14.292568 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:14.292572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:14.292643 (+ 71us) server_negotiation.cc:407] Connection header received
0619 12:30:14.292874 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:14.292878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:14.292985 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:14.293186 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:14.293466 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.293988 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.295378 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.295982 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.296066 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:14.296133 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:14.296470 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 12:30:14.296539 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 12:30:14.296663 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.292225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32903 (local address 127.0.98.1:36373)
0619 12:30:14.292532 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:14.292547 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:14.292709 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:14.293196 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:14.293199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:14.293215 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:14.293341 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:14.293346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.294090 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.294097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:14.295216 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:14.295234 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.296233 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.296236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:14.296344 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:14.296361 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 12:30:14.296456 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 12:30:14.296520 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":19,"spinlock_wait_cycles":36480,"thread_start_us":100,"threads_started":1}
I20260619 12:30:14.296634 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff6fbd952ba45f7b233ddedbe97d846" candidate_uuid: "4ba7985b233a4df2b3819acd024bda3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad34e831164d4a259d12bb31a9c41136" is_pre_election: true
I20260619 12:30:14.296970 4773 raft_consensus.cc:2454] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ba7985b233a4df2b3819acd024bda3e in term 0.
I20260619 12:30:14.297360 4906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff6fbd952ba45f7b233ddedbe97d846" candidate_uuid: "4ba7985b233a4df2b3819acd024bda3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deda46a8042c442cb0229ee0444f61da" is_pre_election: true
I20260619 12:30:14.297506 4574 leader_election.cc:302] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ba7985b233a4df2b3819acd024bda3e, ad34e831164d4a259d12bb31a9c41136; no voters:
I20260619 12:30:14.297633 4906 raft_consensus.cc:2454] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ba7985b233a4df2b3819acd024bda3e in term 0.
I20260619 12:30:14.297812 4976 raft_consensus.cc:2781] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:30:14.297904 4976 raft_consensus.cc:482] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:30:14.297961 4976 raft_consensus.cc:3037] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:14.298990 4976 raft_consensus.cc:504] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad34e831164d4a259d12bb31a9c41136" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "deda46a8042c442cb0229ee0444f61da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32903 } } peers { permanent_uuid: "4ba7985b233a4df2b3819acd024bda3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } }
I20260619 12:30:14.299325 4976 leader_election.cc:288] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [CANDIDATE]: Term 1 election: Requested vote from peers ad34e831164d4a259d12bb31a9c41136 (127.0.98.2:38505), deda46a8042c442cb0229ee0444f61da (127.0.98.3:32903)
I20260619 12:30:14.299683 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff6fbd952ba45f7b233ddedbe97d846" candidate_uuid: "4ba7985b233a4df2b3819acd024bda3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad34e831164d4a259d12bb31a9c41136"
I20260619 12:30:14.299824 4773 raft_consensus.cc:3037] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:14.299823 4906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff6fbd952ba45f7b233ddedbe97d846" candidate_uuid: "4ba7985b233a4df2b3819acd024bda3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deda46a8042c442cb0229ee0444f61da"
I20260619 12:30:14.299997 4906 raft_consensus.cc:3037] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:14.301280 4773 raft_consensus.cc:2454] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ba7985b233a4df2b3819acd024bda3e in term 1.
I20260619 12:30:14.301301 4906 raft_consensus.cc:2454] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ba7985b233a4df2b3819acd024bda3e in term 1.
I20260619 12:30:14.301601 4574 leader_election.cc:302] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ba7985b233a4df2b3819acd024bda3e, ad34e831164d4a259d12bb31a9c41136; no voters:
I20260619 12:30:14.301824 4976 raft_consensus.cc:2781] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:14.302147 4976 raft_consensus.cc:686] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [term 1 LEADER]: Becoming Leader. State: Replica: 4ba7985b233a4df2b3819acd024bda3e, State: Running, Role: LEADER
I20260619 12:30:14.302513 4976 consensus_queue.cc:237] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad34e831164d4a259d12bb31a9c41136" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "deda46a8042c442cb0229ee0444f61da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32903 } } peers { permanent_uuid: "4ba7985b233a4df2b3819acd024bda3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } }
I20260619 12:30:14.305787 4493 catalog_manager.cc:5495] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ba7985b233a4df2b3819acd024bda3e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4ba7985b233a4df2b3819acd024bda3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad34e831164d4a259d12bb31a9c41136" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "deda46a8042c442cb0229ee0444f61da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ba7985b233a4df2b3819acd024bda3e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36721 } health_report { overall_health: HEALTHY } } }
I20260619 12:30:14.322863 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.319266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47920 (local address 127.0.98.1:36721)
0619 12:30:14.319401 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:14.319408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:14.319425 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:30:14.319620 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:14.319623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:14.319677 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:14.319795 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:30:14.320044 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.320556 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.321471 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:14.321693 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:14.321802 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:14.322092 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:30:14.322498 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:30:14.322628 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:14.322667 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 12:30:14.322722 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:30:14.328666 4773 raft_consensus.cc:1261] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136 [term 1 FOLLOWER]: Refusing update from remote peer 4ba7985b233a4df2b3819acd024bda3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:14.328657 4906 raft_consensus.cc:1261] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da [term 1 FOLLOWER]: Refusing update from remote peer 4ba7985b233a4df2b3819acd024bda3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:14.329401 4976 consensus_queue.cc:1035] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "deda46a8042c442cb0229ee0444f61da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:14.329587 4980 consensus_queue.cc:1035] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad34e831164d4a259d12bb31a9c41136" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:14.337304 4985 mvcc.cc:205] Tried to move back new op lower bound from 7298548589883289600 to 7298548589785886720. Current Snapshot: MvccSnapshot[applied={T|T < 7298548589883289600}]
I20260619 12:30:14.345813 4496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:14.346048 4496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:14.348364 4496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 841f409f662e42c79fdcee030835b912: Sending DeleteTablet for 3 replicas of tablet 2ff6fbd952ba45f7b233ddedbe97d846
I20260619 12:30:14.349148 4620 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff6fbd952ba45f7b233ddedbe97d846 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47918
I20260619 12:30:14.349188 4886 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff6fbd952ba45f7b233ddedbe97d846 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42226
I20260619 12:30:14.349401 4753 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff6fbd952ba45f7b233ddedbe97d846 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54680
I20260619 12:30:14.349634 4994 tablet_replica.cc:330] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e: stopping tablet replica
I20260619 12:30:14.349684 4993 tablet_replica.cc:330] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da: stopping tablet replica
I20260619 12:30:14.349927 4993 raft_consensus.cc:2229] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:14.349922 4994 raft_consensus.cc:2229] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:30:14.350208 4993 raft_consensus.cc:2258] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:14.350314 4994 raft_consensus.cc:2258] T 2ff6fbd952ba45f7b233ddedbe97d846 P 4ba7985b233a4df2b3819acd024bda3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:14.350857 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 4558
I20260619 12:30:14.351771 4993 ts_tablet_manager.cc:1905] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:14.351837 4995 tablet_replica.cc:330] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136: stopping tablet replica
I20260619 12:30:14.352087 4995 raft_consensus.cc:2229] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:14.352355 4995 raft_consensus.cc:2258] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:14.353782 4995 ts_tablet_manager.cc:1905] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:14.354504 4993 ts_tablet_manager.cc:1918] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:30:14.354601 4993 log.cc:1196] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/2ff6fbd952ba45f7b233ddedbe97d846
I20260619 12:30:14.354983 4993 ts_tablet_manager.cc:1939] T 2ff6fbd952ba45f7b233ddedbe97d846 P deda46a8042c442cb0229ee0444f61da: Deleting consensus metadata
I20260619 12:30:14.356200 4481 catalog_manager.cc:4841] TS deda46a8042c442cb0229ee0444f61da (127.0.98.3:32903): tablet 2ff6fbd952ba45f7b233ddedbe97d846 (table test-table [id=b0e11ebe77904850929bf3a20d8fdb80]) successfully deleted
I20260619 12:30:14.356580 4995 ts_tablet_manager.cc:1918] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:30:14.356688 4995 log.cc:1196] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/2ff6fbd952ba45f7b233ddedbe97d846
I20260619 12:30:14.356984 4995 ts_tablet_manager.cc:1939] T 2ff6fbd952ba45f7b233ddedbe97d846 P ad34e831164d4a259d12bb31a9c41136: Deleting consensus metadata
W20260619 12:30:14.357860 4480 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36721 (error 108)
I20260619 12:30:14.358088 4481 catalog_manager.cc:4841] TS ad34e831164d4a259d12bb31a9c41136 (127.0.98.2:38505): tablet 2ff6fbd952ba45f7b233ddedbe97d846 (table test-table [id=b0e11ebe77904850929bf3a20d8fdb80]) successfully deleted
I20260619 12:30:14.358605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 4691
I20260619 12:30:14.359006 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.358642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36721 (local address 127.0.0.1:47922)
0619 12:30:14.358846 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:14.358916 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260619 12:30:14.359169 4480 catalog_manager.cc:4584] TS 4ba7985b233a4df2b3819acd024bda3e (127.0.98.1:36721): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ff6fbd952ba45f7b233ddedbe97d846: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36721: connect: Connection refused (error 111)
I20260619 12:30:14.363996 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 4824
I20260619 12:30:14.369309 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 4465
2026-06-19T12:30:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2310 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T12:30:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:30:14Z Disabled control of system clock
I20260619 12:30:14.398675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40041
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 12:30:14.507609 5000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:14.507879 5000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:14.507933 5000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:14.512194 5000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:14.512262 5000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:14.512279 5000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:14.512296 5000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:14.512311 5000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:30:14.517241 5000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:14.518262 5000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:14.519596 5000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:14.527868 5008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:14.527871 5005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:14.527917 5006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:14.528285 5000 server_base.cc:1034] running on GCE node
I20260619 12:30:14.528842 5000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:14.530002 5000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:14.531199 5000 hybrid_clock.cc:648] HybridClock initialized: now 1781872214531172 us; error 47 us; skew 500 ppm
I20260619 12:30:14.533049 5000 webserver.cc:460] Webserver started at http://127.0.98.62:42903/ using document root <none> and password file <none>
I20260619 12:30:14.533728 5000 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:14.533828 5000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:14.534061 5000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:14.536109 5000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "3cdb853503014e2d8c6beb2781771502"
format_stamp: "Formatted at 2026-06-19 12:30:14 on dist-test-slave-fg3h"
server_key: "7ba7f4c25eb95fc56d8554e71c9ae4c1"
server_key_iv: "66d9e5357e1f1c3948940acd3400a9fa"
server_key_version: "encryptionkey@0"
I20260619 12:30:14.536669 5000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "3cdb853503014e2d8c6beb2781771502"
format_stamp: "Formatted at 2026-06-19 12:30:14 on dist-test-slave-fg3h"
server_key: "7ba7f4c25eb95fc56d8554e71c9ae4c1"
server_key_iv: "66d9e5357e1f1c3948940acd3400a9fa"
server_key_version: "encryptionkey@0"
I20260619 12:30:14.540426 5000 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 12:30:14.542958 5014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:14.544278 5000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:30:14.544440 5000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "3cdb853503014e2d8c6beb2781771502"
format_stamp: "Formatted at 2026-06-19 12:30:14 on dist-test-slave-fg3h"
server_key: "7ba7f4c25eb95fc56d8554e71c9ae4c1"
server_key_iv: "66d9e5357e1f1c3948940acd3400a9fa"
server_key_version: "encryptionkey@0"
I20260619 12:30:14.544566 5000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:14.549211 5000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:14.549852 5000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:14.550057 5000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:14.556600 5000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40041
I20260619 12:30:14.556627 5066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40041 every 8 connection(s)
I20260619 12:30:14.557548 5000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:30:14.560657 5067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:14.564028 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 5000
I20260619 12:30:14.564177 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:30:14.564459 392 external_mini_cluster.cc:1428] Setting key 518ddee8749375ef47af7ecd36b0ceeb
I20260619 12:30:14.566453 5067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502: Bootstrap starting.
I20260619 12:30:14.568759 5067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:14.569684 5067 log.cc:824] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:14.570374 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.566061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.0.98.62:40041)
0619 12:30:14.566718 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:14.566735 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:14.566771 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:30:14.567460 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:14.567477 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:14.567826 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:14.568112 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:14.568988 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:14.568994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:14.569015 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:14.569026 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:14.569128 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:14.569152 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:14.569256 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:14.569385 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 12:30:14.569667 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":136,"threads_started":1}
I20260619 12:30:14.571681 5067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502: No bootstrap required, opened a new log
I20260619 12:30:14.575043 5067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdb853503014e2d8c6beb2781771502" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40041 } }
I20260619 12:30:14.575239 5067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:14.575310 5067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cdb853503014e2d8c6beb2781771502, State: Initialized, Role: FOLLOWER
I20260619 12:30:14.575743 5067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdb853503014e2d8c6beb2781771502" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40041 } }
I20260619 12:30:14.575891 5067 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:30:14.575965 5067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:30:14.576068 5067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:14.576907 5067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdb853503014e2d8c6beb2781771502" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40041 } }
I20260619 12:30:14.577255 5067 leader_election.cc:302] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cdb853503014e2d8c6beb2781771502; no voters:
I20260619 12:30:14.577543 5067 leader_election.cc:288] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:30:14.577692 5072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:14.577924 5072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502 [term 1 LEADER]: Becoming Leader. State: Replica: 3cdb853503014e2d8c6beb2781771502, State: Running, Role: LEADER
I20260619 12:30:14.578234 5072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdb853503014e2d8c6beb2781771502" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40041 } }
I20260619 12:30:14.578682 5067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:14.579902 5074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cdb853503014e2d8c6beb2781771502. Latest consensus state: current_term: 1 leader_uuid: "3cdb853503014e2d8c6beb2781771502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdb853503014e2d8c6beb2781771502" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40041 } } }
I20260619 12:30:14.579902 5073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cdb853503014e2d8c6beb2781771502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdb853503014e2d8c6beb2781771502" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40041 } } }
I20260619 12:30:14.580135 5073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:14.580164 5074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:14.580541 5082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:30:14.583309 5082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:30:14.588537 5082 catalog_manager.cc:1347] Generated new cluster ID: f7db6ea801fd47bcb58662b8d3b72de3
I20260619 12:30:14.588600 5082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:30:14.610894 5082 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:30:14.611223 5082 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:30:14.616361 5082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502: Generated new TSK 0
I20260619 12:30:14.617046 5082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:30:14.620600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40041
--builtin_ntp_servers=127.0.98.20:35983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 12:30:14.732884 5091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:14.733227 5091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:14.733273 5091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:14.737401 5091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:14.737455 5091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:14.737514 5091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:30:14.742364 5091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:14.743392 5091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:14.744715 5091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:14.754299 5097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:14.754499 5091 server_base.cc:1034] running on GCE node
W20260619 12:30:14.754354 5099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:14.754305 5096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:14.755033 5091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:14.755733 5091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:14.756896 5091 hybrid_clock.cc:648] HybridClock initialized: now 1781872214756863 us; error 45 us; skew 500 ppm
I20260619 12:30:14.758904 5091 webserver.cc:460] Webserver started at http://127.0.98.1:32983/ using document root <none> and password file <none>
I20260619 12:30:14.759495 5091 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:14.759558 5091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:14.759773 5091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:14.761879 5091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "cfd6d0cf4da54309b260c810e1998063"
format_stamp: "Formatted at 2026-06-19 12:30:14 on dist-test-slave-fg3h"
server_key: "630e68edc8d78110c8f8b66a4f84fb64"
server_key_iv: "cbbdea2cf2045169a691c2f18a2dcb10"
server_key_version: "encryptionkey@0"
I20260619 12:30:14.762439 5091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cfd6d0cf4da54309b260c810e1998063"
format_stamp: "Formatted at 2026-06-19 12:30:14 on dist-test-slave-fg3h"
server_key: "630e68edc8d78110c8f8b66a4f84fb64"
server_key_iv: "cbbdea2cf2045169a691c2f18a2dcb10"
server_key_version: "encryptionkey@0"
I20260619 12:30:14.766194 5091 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:30:14.768647 5105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:14.769960 5091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:14.770104 5091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "cfd6d0cf4da54309b260c810e1998063"
format_stamp: "Formatted at 2026-06-19 12:30:14 on dist-test-slave-fg3h"
server_key: "630e68edc8d78110c8f8b66a4f84fb64"
server_key_iv: "cbbdea2cf2045169a691c2f18a2dcb10"
server_key_version: "encryptionkey@0"
I20260619 12:30:14.770221 5091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:14.774726 5091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:14.775310 5091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:14.775497 5091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:14.776031 5091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:14.776902 5091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:14.776976 5091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:14.777045 5091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:14.777114 5091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:14.786049 5091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40481
I20260619 12:30:14.786078 5218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40481 every 8 connection(s)
I20260619 12:30:14.786962 5091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:30:14.793231 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.788760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49511 (local address 127.0.98.62:40041)
0619 12:30:14.788907 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:14.788911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:14.790104 (+ 1193us) server_negotiation.cc:407] Connection header received
0619 12:30:14.791407 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:14.791409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:14.791446 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:14.791545 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:14.792645 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:14.792648 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:14.792650 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:14.792661 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:14.792678 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:14.792686 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:14.792766 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:14.792996 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 12:30:14.793043 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:30:14.794091 5221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.789293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40041 (local address 127.0.98.1:49511)
0619 12:30:14.789959 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:14.789998 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:14.791212 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:14.791725 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:14.791732 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:14.791819 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:14.792377 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:30:14.792378 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:14.792419 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:30:14.792498 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:14.792785 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:14.792792 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:14.792797 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:30:14.792976 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 12:30:14.793357 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
I20260619 12:30:14.795380 5219 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40041
I20260619 12:30:14.795603 5219 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:14.796039 5219 heartbeater.cc:511] Master 127.0.98.62:40041 requested a full tablet report, sending...
I20260619 12:30:14.796720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 5091
I20260619 12:30:14.796883 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:30:14.797204 392 external_mini_cluster.cc:1428] Setting key 492442c7e2fdab3ae2d29c4065aed14e
I20260619 12:30:14.797663 5031 ts_manager.cc:194] Registered new tserver with Master: cfd6d0cf4da54309b260c810e1998063 (127.0.98.1:40481)
I20260619 12:30:14.799223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40041
--builtin_ntp_servers=127.0.98.20:35983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 12:30:14.909874 5222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:14.910204 5222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:14.910281 5222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:14.914320 5222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:14.914414 5222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:14.914508 5222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:30:14.919211 5222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:14.920308 5222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:14.921717 5222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:14.931166 5227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:14.931136 5228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:14.931156 5230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:14.931263 5222 server_base.cc:1034] running on GCE node
I20260619 12:30:14.931782 5222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:14.932415 5222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:14.933575 5222 hybrid_clock.cc:648] HybridClock initialized: now 1781872214933543 us; error 45 us; skew 500 ppm
I20260619 12:30:14.935511 5222 webserver.cc:460] Webserver started at http://127.0.98.2:45017/ using document root <none> and password file <none>
I20260619 12:30:14.936082 5222 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:14.936144 5222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:14.936386 5222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:14.938511 5222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "5ccd9f737bfe481f9cc6a0c7b629ecfc"
format_stamp: "Formatted at 2026-06-19 12:30:14 on dist-test-slave-fg3h"
server_key: "3ae4437764aed845fbac20021dda70e8"
server_key_iv: "0fa634c63632c31708b08e67fbd8721f"
server_key_version: "encryptionkey@0"
I20260619 12:30:14.939095 5222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5ccd9f737bfe481f9cc6a0c7b629ecfc"
format_stamp: "Formatted at 2026-06-19 12:30:14 on dist-test-slave-fg3h"
server_key: "3ae4437764aed845fbac20021dda70e8"
server_key_iv: "0fa634c63632c31708b08e67fbd8721f"
server_key_version: "encryptionkey@0"
I20260619 12:30:14.942785 5222 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 12:30:14.945276 5236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:14.946467 5222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:30:14.946609 5222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "5ccd9f737bfe481f9cc6a0c7b629ecfc"
format_stamp: "Formatted at 2026-06-19 12:30:14 on dist-test-slave-fg3h"
server_key: "3ae4437764aed845fbac20021dda70e8"
server_key_iv: "0fa634c63632c31708b08e67fbd8721f"
server_key_version: "encryptionkey@0"
I20260619 12:30:14.946724 5222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:14.951799 5222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:14.952400 5222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:14.952592 5222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:14.953207 5222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:14.954113 5222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:14.954185 5222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:14.954254 5222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:14.954306 5222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:14.963200 5222 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36333
I20260619 12:30:14.963228 5349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36333 every 8 connection(s)
I20260619 12:30:14.964107 5222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:30:14.965473 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 5222
I20260619 12:30:14.965598 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:30:14.965878 392 external_mini_cluster.cc:1428] Setting key 10ce695d4e84f26fd1860a2837f05ac2
I20260619 12:30:14.967963 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40041
--builtin_ntp_servers=127.0.98.20:35983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 12:30:14.970928 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.967414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50367 (local address 127.0.98.62:40041)
0619 12:30:14.967524 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:14.967529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:14.967543 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:30:14.968925 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:14.968931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:14.968981 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:14.969115 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:14.970446 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:14.970449 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:14.970451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:14.970462 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:14.970481 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:14.970489 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:14.970565 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:14.970749 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 12:30:14.970796 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 12:30:14.971804 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:14.966651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40041 (local address 127.0.98.2:50367)
0619 12:30:14.967408 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:14.967462 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:14.968710 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:14.969334 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:14.969346 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:14.969507 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:14.970113 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:30:14.970117 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:14.970178 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:30:14.970273 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:14.970573 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:14.970580 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:14.970584 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:30:14.970761 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 12:30:14.971048 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":155,"threads_started":1}
I20260619 12:30:14.972947 5350 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40041
I20260619 12:30:14.973210 5350 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:14.973639 5350 heartbeater.cc:511] Master 127.0.98.62:40041 requested a full tablet report, sending...
I20260619 12:30:14.974771 5031 ts_manager.cc:194] Registered new tserver with Master: 5ccd9f737bfe481f9cc6a0c7b629ecfc (127.0.98.2:36333)
W20260619 12:30:15.083962 5353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:15.084215 5353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:15.084254 5353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:15.088260 5353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:15.088315 5353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:15.088369 5353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:30:15.093254 5353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:15.094331 5353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:15.095669 5353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:15.105312 5359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:15.105382 5353 server_base.cc:1034] running on GCE node
W20260619 12:30:15.105369 5361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:15.105332 5358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:15.106027 5353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:15.106693 5353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:15.107882 5353 hybrid_clock.cc:648] HybridClock initialized: now 1781872215107865 us; error 29 us; skew 500 ppm
I20260619 12:30:15.109884 5353 webserver.cc:460] Webserver started at http://127.0.98.3:46075/ using document root <none> and password file <none>
I20260619 12:30:15.110473 5353 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:15.110571 5353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:15.110795 5353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:15.112848 5353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "f2ae13a7e53446aebfa24760c86bf85e"
format_stamp: "Formatted at 2026-06-19 12:30:15 on dist-test-slave-fg3h"
server_key: "dbc9f242a4ef85470934ff437937aba7"
server_key_iv: "2e1058118d34aa24ecd6e4fc2f290c72"
server_key_version: "encryptionkey@0"
I20260619 12:30:15.113483 5353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f2ae13a7e53446aebfa24760c86bf85e"
format_stamp: "Formatted at 2026-06-19 12:30:15 on dist-test-slave-fg3h"
server_key: "dbc9f242a4ef85470934ff437937aba7"
server_key_iv: "2e1058118d34aa24ecd6e4fc2f290c72"
server_key_version: "encryptionkey@0"
I20260619 12:30:15.117333 5353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:30:15.119740 5367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:15.120926 5353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:30:15.121129 5353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "f2ae13a7e53446aebfa24760c86bf85e"
format_stamp: "Formatted at 2026-06-19 12:30:15 on dist-test-slave-fg3h"
server_key: "dbc9f242a4ef85470934ff437937aba7"
server_key_iv: "2e1058118d34aa24ecd6e4fc2f290c72"
server_key_version: "encryptionkey@0"
I20260619 12:30:15.121285 5353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:15.126210 5353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:15.126796 5353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:15.126991 5353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:15.127529 5353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:15.128407 5353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:15.128477 5353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:15.128567 5353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:15.128616 5353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:15.137550 5353 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46357
I20260619 12:30:15.137573 5480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46357 every 8 connection(s)
I20260619 12:30:15.138427 5353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:30:15.144464 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:15.140271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60441 (local address 127.0.98.62:40041)
0619 12:30:15.140401 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:15.140405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:15.141502 (+ 1097us) server_negotiation.cc:407] Connection header received
0619 12:30:15.142780 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:15.142783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:15.142826 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:15.142928 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:15.143985 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:15.143989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:15.143992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:15.144005 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:15.144027 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:15.144042 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:15.144137 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:15.144285 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 12:30:15.144330 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 12:30:15.144934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 5353
I20260619 12:30:15.145063 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:30:15.145364 392 external_mini_cluster.cc:1428] Setting key f1e3d8688ec5af6d231ed569531d818d
I20260619 12:30:15.145481 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:15.140660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40041 (local address 127.0.98.3:60441)
0619 12:30:15.141353 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:15.141394 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:15.142575 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:15.143096 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:15.143102 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:15.143189 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:15.143734 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:30:15.143735 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:15.143773 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:30:15.143839 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:15.144131 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:15.144139 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:15.144146 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 12:30:15.144319 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 12:30:15.144552 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
I20260619 12:30:15.146659 5481 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40041
I20260619 12:30:15.146884 5481 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:15.147367 5481 heartbeater.cc:511] Master 127.0.98.62:40041 requested a full tablet report, sending...
I20260619 12:30:15.148386 5031 ts_manager.cc:194] Registered new tserver with Master: f2ae13a7e53446aebfa24760c86bf85e (127.0.98.3:46357)
I20260619 12:30:15.159163 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:30:15.162515 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:15.161149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.0.98.62:40041)
0619 12:30:15.161282 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:15.161286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:15.161406 (+ 120us) server_negotiation.cc:407] Connection header received
0619 12:30:15.161501 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:15.161503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:15.161536 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:15.161598 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:15.162227 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:15.162230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:15.162232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:15.162242 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:15.162260 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:15.162267 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:15.162330 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:15.162385 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 12:30:15.162420 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 12:30:15.165477 5031 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:30:15.168473 5031 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:30:15.180034 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:15.176970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40481 (local address 127.0.0.1:60430)
0619 12:30:15.177758 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:15.177773 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:15.177854 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:15.178326 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:15.178332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:15.178414 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:15.179405 (+ 991us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:30:15.179407 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:15.179416 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:30:15.179444 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:15.179765 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:15.179768 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:15.179771 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:30:15.179830 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 12:30:15.179911 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":43,"thread_start_us":592,"threads_started":1}
I20260619 12:30:15.180187 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:15.177316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.0.98.1:40481)
0619 12:30:15.177728 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:15.177740 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:15.177850 (+ 110us) server_negotiation.cc:407] Connection header received
0619 12:30:15.177949 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:15.177955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:15.178154 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:15.178717 (+ 563us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:15.179572 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:15.179579 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:15.179591 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:15.179602 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:15.179648 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:15.179684 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:15.179800 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:15.179893 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 12:30:15.179979 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":123,"threads_started":1}
I20260619 12:30:15.180564 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:15.176912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46357 (local address 127.0.0.1:40490)
0619 12:30:15.177374 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:15.177432 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:15.177545 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:15.178326 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:15.178332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:15.178414 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:15.179217 (+ 803us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:30:15.179219 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:15.179239 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:30:15.179307 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:15.179659 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:15.179669 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:15.179678 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 12:30:15.179771 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 12:30:15.179875 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":93,"threads_started":1}
I20260619 12:30:15.180747 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:15.177126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40490 (local address 127.0.98.3:46357)
0619 12:30:15.177925 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:15.177936 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:15.177954 (+ 18us) server_negotiation.cc:407] Connection header received
0619 12:30:15.177998 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:15.178002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:15.178154 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:15.178717 (+ 563us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:15.179423 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:15.179429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:15.179446 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:15.179476 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:15.179517 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:15.179544 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:15.180473 (+ 929us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:15.180534 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 12:30:15.180598 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":153,"threads_started":1}
I20260619 12:30:15.180990 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:15.177836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36333 (local address 127.0.0.1:51118)
0619 12:30:15.178195 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:15.178210 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:15.178286 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:15.179522 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:15.179525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:15.179542 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:15.180323 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:30:15.180324 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:15.180334 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:30:15.180360 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:15.180749 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:15.180754 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:15.180757 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:30:15.180820 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 12:30:15.180862 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":86,"threads_started":1}
I20260619 12:30:15.181515 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:15.178131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.0.98.2:36333)
0619 12:30:15.178523 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:15.178544 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:15.178568 (+ 24us) server_negotiation.cc:407] Connection header received
0619 12:30:15.178622 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:15.178628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:15.179384 (+ 756us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:15.179535 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:15.180539 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:15.180545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:15.180556 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:15.180567 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:15.180609 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:15.180639 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:15.181255 (+ 616us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:15.181300 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 12:30:15.181363 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":99,"threads_started":1}
I20260619 12:30:15.183205 5153 tablet_service.cc:1467] Processing CreateTablet for tablet b0bdf775e97f42b5ba4a04aedf23ae49 (DEFAULT_TABLE table=test-table [id=e6825f313fcf416eba754cf0e4018806]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:15.183302 5415 tablet_service.cc:1467] Processing CreateTablet for tablet b0bdf775e97f42b5ba4a04aedf23ae49 (DEFAULT_TABLE table=test-table [id=e6825f313fcf416eba754cf0e4018806]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:15.184075 5153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0bdf775e97f42b5ba4a04aedf23ae49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:15.183861 5284 tablet_service.cc:1467] Processing CreateTablet for tablet b0bdf775e97f42b5ba4a04aedf23ae49 (DEFAULT_TABLE table=test-table [id=e6825f313fcf416eba754cf0e4018806]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:15.184268 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0bdf775e97f42b5ba4a04aedf23ae49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:15.184733 5284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0bdf775e97f42b5ba4a04aedf23ae49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:15.190168 5498 tablet_bootstrap.cc:492] T b0bdf775e97f42b5ba4a04aedf23ae49 P cfd6d0cf4da54309b260c810e1998063: Bootstrap starting.
I20260619 12:30:15.191288 5499 tablet_bootstrap.cc:492] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e: Bootstrap starting.
I20260619 12:30:15.191809 5500 tablet_bootstrap.cc:492] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc: Bootstrap starting.
I20260619 12:30:15.192559 5498 tablet_bootstrap.cc:654] T b0bdf775e97f42b5ba4a04aedf23ae49 P cfd6d0cf4da54309b260c810e1998063: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:15.193454 5498 log.cc:824] T b0bdf775e97f42b5ba4a04aedf23ae49 P cfd6d0cf4da54309b260c810e1998063: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:15.193507 5499 tablet_bootstrap.cc:654] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:15.193598 5500 tablet_bootstrap.cc:654] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:15.194291 5499 log.cc:824] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:15.194406 5500 log.cc:824] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:15.195571 5498 tablet_bootstrap.cc:492] T b0bdf775e97f42b5ba4a04aedf23ae49 P cfd6d0cf4da54309b260c810e1998063: No bootstrap required, opened a new log
I20260619 12:30:15.195757 5498 ts_tablet_manager.cc:1397] T b0bdf775e97f42b5ba4a04aedf23ae49 P cfd6d0cf4da54309b260c810e1998063: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:30:15.196048 5499 tablet_bootstrap.cc:492] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e: No bootstrap required, opened a new log
I20260619 12:30:15.196247 5499 ts_tablet_manager.cc:1397] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 12:30:15.196302 5500 tablet_bootstrap.cc:492] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc: No bootstrap required, opened a new log
I20260619 12:30:15.196478 5500 ts_tablet_manager.cc:1397] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 12:30:15.199832 5499 raft_consensus.cc:348] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae13a7e53446aebfa24760c86bf85e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46357 } } peers { permanent_uuid: "cfd6d0cf4da54309b260c810e1998063" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40481 } } peers { permanent_uuid: "5ccd9f737bfe481f9cc6a0c7b629ecfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } }
I20260619 12:30:15.200070 5499 raft_consensus.cc:374] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:15.200122 5499 raft_consensus.cc:729] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2ae13a7e53446aebfa24760c86bf85e, State: Initialized, Role: FOLLOWER
I20260619 12:30:15.200325 5498 raft_consensus.cc:348] T b0bdf775e97f42b5ba4a04aedf23ae49 P cfd6d0cf4da54309b260c810e1998063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae13a7e53446aebfa24760c86bf85e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46357 } } peers { permanent_uuid: "cfd6d0cf4da54309b260c810e1998063" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40481 } } peers { permanent_uuid: "5ccd9f737bfe481f9cc6a0c7b629ecfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } }
I20260619 12:30:15.200592 5498 raft_consensus.cc:374] T b0bdf775e97f42b5ba4a04aedf23ae49 P cfd6d0cf4da54309b260c810e1998063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:15.200659 5498 raft_consensus.cc:729] T b0bdf775e97f42b5ba4a04aedf23ae49 P cfd6d0cf4da54309b260c810e1998063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd6d0cf4da54309b260c810e1998063, State: Initialized, Role: FOLLOWER
I20260619 12:30:15.200598 5499 consensus_queue.cc:260] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae13a7e53446aebfa24760c86bf85e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46357 } } peers { permanent_uuid: "cfd6d0cf4da54309b260c810e1998063" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40481 } } peers { permanent_uuid: "5ccd9f737bfe481f9cc6a0c7b629ecfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } }
I20260619 12:30:15.200835 5500 raft_consensus.cc:348] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae13a7e53446aebfa24760c86bf85e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46357 } } peers { permanent_uuid: "cfd6d0cf4da54309b260c810e1998063" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40481 } } peers { permanent_uuid: "5ccd9f737bfe481f9cc6a0c7b629ecfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } }
I20260619 12:30:15.201062 5500 raft_consensus.cc:374] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:15.201152 5500 raft_consensus.cc:729] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ccd9f737bfe481f9cc6a0c7b629ecfc, State: Initialized, Role: FOLLOWER
I20260619 12:30:15.201097 5498 consensus_queue.cc:260] T b0bdf775e97f42b5ba4a04aedf23ae49 P cfd6d0cf4da54309b260c810e1998063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae13a7e53446aebfa24760c86bf85e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46357 } } peers { permanent_uuid: "cfd6d0cf4da54309b260c810e1998063" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40481 } } peers { permanent_uuid: "5ccd9f737bfe481f9cc6a0c7b629ecfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } }
I20260619 12:30:15.201431 5481 heartbeater.cc:503] Master 127.0.98.62:40041 was elected leader, sending a full tablet report...
I20260619 12:30:15.201664 5499 ts_tablet_manager.cc:1428] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 12:30:15.201601 5500 consensus_queue.cc:260] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae13a7e53446aebfa24760c86bf85e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46357 } } peers { permanent_uuid: "cfd6d0cf4da54309b260c810e1998063" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40481 } } peers { permanent_uuid: "5ccd9f737bfe481f9cc6a0c7b629ecfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } }
I20260619 12:30:15.202117 5498 ts_tablet_manager.cc:1428] T b0bdf775e97f42b5ba4a04aedf23ae49 P cfd6d0cf4da54309b260c810e1998063: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 12:30:15.202183 5350 heartbeater.cc:503] Master 127.0.98.62:40041 was elected leader, sending a full tablet report...
I20260619 12:30:15.202415 5219 heartbeater.cc:503] Master 127.0.98.62:40041 was elected leader, sending a full tablet report...
I20260619 12:30:15.203413 5500 ts_tablet_manager.cc:1428] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260619 12:30:15.215035 5351 tablet.cc:2369] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:30:15.288162 5220 tablet.cc:2369] T b0bdf775e97f42b5ba4a04aedf23ae49 P cfd6d0cf4da54309b260c810e1998063: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:30:15.315280 5504 raft_consensus.cc:482] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:30:15.315536 5504 raft_consensus.cc:504] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae13a7e53446aebfa24760c86bf85e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46357 } } peers { permanent_uuid: "cfd6d0cf4da54309b260c810e1998063" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40481 } } peers { permanent_uuid: "5ccd9f737bfe481f9cc6a0c7b629ecfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } }
I20260619 12:30:15.316627 5504 leader_election.cc:288] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfd6d0cf4da54309b260c810e1998063 (127.0.98.1:40481), 5ccd9f737bfe481f9cc6a0c7b629ecfc (127.0.98.2:36333)
I20260619 12:30:15.318612 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:15.316906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40481 (local address 127.0.98.3:50369)
0619 12:30:15.317032 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:15.317051 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:15.317183 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:15.317447 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:15.317450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:15.317465 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:15.318061 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:30:15.318063 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:15.318077 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:30:15.318105 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:15.318373 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:15.318376 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:15.318379 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 12:30:15.318453 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 12:30:15.318489 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 12:30:15.318612 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:15.317022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50369 (local address 127.0.98.1:40481)
0619 12:30:15.317184 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:15.317191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:15.317210 (+ 19us) server_negotiation.cc:407] Connection header received
0619 12:30:15.317268 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:15.317270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:15.317311 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:15.317413 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:15.318241 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:15.318245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:15.318247 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:15.318259 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:15.318277 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:15.318285 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:15.318379 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:15.318451 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 12:30:15.318489 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 12:30:15.319048 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:15.317369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50051 (local address 127.0.98.2:36333)
0619 12:30:15.317505 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:15.317512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:15.317763 (+ 251us) server_negotiation.cc:407] Connection header received
0619 12:30:15.317850 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:15.317875 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:15.317925 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:15.318023 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:15.318697 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:15.318703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:15.318706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:15.318722 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:15.318748 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:15.318760 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:15.318837 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:15.318903 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 12:30:15.318940 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 12:30:15.319068 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:15.317242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36333 (local address 127.0.98.3:50051)
0619 12:30:15.317656 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:15.317672 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:15.317767 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:15.318061 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:15.318063 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:15.318076 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:15.318530 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:30:15.318533 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:15.318547 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:30:15.318570 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:15.318830 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:15.318833 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:15.318835 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 12:30:15.318906 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 12:30:15.318939 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":147,"threads_started":1}
I20260619 12:30:15.319294 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0bdf775e97f42b5ba4a04aedf23ae49" candidate_uuid: "f2ae13a7e53446aebfa24760c86bf85e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfd6d0cf4da54309b260c810e1998063" is_pre_election: true
I20260619 12:30:15.319587 5173 raft_consensus.cc:2454] T b0bdf775e97f42b5ba4a04aedf23ae49 P cfd6d0cf4da54309b260c810e1998063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2ae13a7e53446aebfa24760c86bf85e in term 0.
I20260619 12:30:15.319571 5304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0bdf775e97f42b5ba4a04aedf23ae49" candidate_uuid: "f2ae13a7e53446aebfa24760c86bf85e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ccd9f737bfe481f9cc6a0c7b629ecfc" is_pre_election: true
I20260619 12:30:15.319831 5304 raft_consensus.cc:2454] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2ae13a7e53446aebfa24760c86bf85e in term 0.
I20260619 12:30:15.320029 5370 leader_election.cc:302] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfd6d0cf4da54309b260c810e1998063, f2ae13a7e53446aebfa24760c86bf85e; no voters:
I20260619 12:30:15.320303 5504 raft_consensus.cc:2781] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:30:15.320376 5504 raft_consensus.cc:482] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:30:15.320406 5504 raft_consensus.cc:3037] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:15.321532 5504 raft_consensus.cc:504] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae13a7e53446aebfa24760c86bf85e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46357 } } peers { permanent_uuid: "cfd6d0cf4da54309b260c810e1998063" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40481 } } peers { permanent_uuid: "5ccd9f737bfe481f9cc6a0c7b629ecfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } }
I20260619 12:30:15.321862 5504 leader_election.cc:288] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [CANDIDATE]: Term 1 election: Requested vote from peers cfd6d0cf4da54309b260c810e1998063 (127.0.98.1:40481), 5ccd9f737bfe481f9cc6a0c7b629ecfc (127.0.98.2:36333)
I20260619 12:30:15.322305 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0bdf775e97f42b5ba4a04aedf23ae49" candidate_uuid: "f2ae13a7e53446aebfa24760c86bf85e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfd6d0cf4da54309b260c810e1998063"
I20260619 12:30:15.322362 5304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0bdf775e97f42b5ba4a04aedf23ae49" candidate_uuid: "f2ae13a7e53446aebfa24760c86bf85e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ccd9f737bfe481f9cc6a0c7b629ecfc"
I20260619 12:30:15.322439 5173 raft_consensus.cc:3037] T b0bdf775e97f42b5ba4a04aedf23ae49 P cfd6d0cf4da54309b260c810e1998063 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:15.322533 5304 raft_consensus.cc:3037] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:15.323503 5173 raft_consensus.cc:2454] T b0bdf775e97f42b5ba4a04aedf23ae49 P cfd6d0cf4da54309b260c810e1998063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2ae13a7e53446aebfa24760c86bf85e in term 1.
I20260619 12:30:15.323582 5304 raft_consensus.cc:2454] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2ae13a7e53446aebfa24760c86bf85e in term 1.
I20260619 12:30:15.323808 5370 leader_election.cc:302] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfd6d0cf4da54309b260c810e1998063, f2ae13a7e53446aebfa24760c86bf85e; no voters:
I20260619 12:30:15.323984 5504 raft_consensus.cc:2781] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:15.324254 5504 raft_consensus.cc:686] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [term 1 LEADER]: Becoming Leader. State: Replica: f2ae13a7e53446aebfa24760c86bf85e, State: Running, Role: LEADER
I20260619 12:30:15.324616 5504 consensus_queue.cc:237] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae13a7e53446aebfa24760c86bf85e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46357 } } peers { permanent_uuid: "cfd6d0cf4da54309b260c810e1998063" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40481 } } peers { permanent_uuid: "5ccd9f737bfe481f9cc6a0c7b629ecfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } }
I20260619 12:30:15.327920 5029 catalog_manager.cc:5495] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e reported cstate change: term changed from 0 to 1, leader changed from <none> to f2ae13a7e53446aebfa24760c86bf85e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f2ae13a7e53446aebfa24760c86bf85e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2ae13a7e53446aebfa24760c86bf85e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfd6d0cf4da54309b260c810e1998063" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ccd9f737bfe481f9cc6a0c7b629ecfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } health_report { overall_health: UNKNOWN } } }
I20260619 12:30:15.338402 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:15.336960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40502 (local address 127.0.98.3:46357)
0619 12:30:15.337125 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:15.337130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:15.337150 (+ 20us) server_negotiation.cc:407] Connection header received
0619 12:30:15.337296 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:15.337299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:15.337351 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:15.337420 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:15.338115 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:15.338118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:15.338120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:15.338128 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:15.338144 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:15.338151 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:15.338221 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:15.338276 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 12:30:15.338306 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:30:15.344245 5173 raft_consensus.cc:1261] T b0bdf775e97f42b5ba4a04aedf23ae49 P cfd6d0cf4da54309b260c810e1998063 [term 1 FOLLOWER]: Refusing update from remote peer f2ae13a7e53446aebfa24760c86bf85e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:15.344249 5304 raft_consensus.cc:1261] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc [term 1 FOLLOWER]: Refusing update from remote peer f2ae13a7e53446aebfa24760c86bf85e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:15.344995 5504 consensus_queue.cc:1035] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfd6d0cf4da54309b260c810e1998063" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:15.345257 5508 consensus_queue.cc:1035] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ccd9f737bfe481f9cc6a0c7b629ecfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:15.353297 5512 mvcc.cc:205] Tried to move back new op lower bound from 7298548594042437632 to 7298548593972649984. Current Snapshot: MvccSnapshot[applied={T|T < 7298548594042437632}]
I20260619 12:30:15.355472 5513 mvcc.cc:205] Tried to move back new op lower bound from 7298548594042437632 to 7298548593972649984. Current Snapshot: MvccSnapshot[applied={T|T < 7298548594042437632}]
I20260619 12:30:15.361071 5029 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:15.361289 5029 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:15.363780 5029 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3cdb853503014e2d8c6beb2781771502: Sending DeleteTablet for 3 replicas of tablet b0bdf775e97f42b5ba4a04aedf23ae49
I20260619 12:30:15.364544 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet b0bdf775e97f42b5ba4a04aedf23ae49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:15 UTC) from {username='slave'} at 127.0.0.1:40490
I20260619 12:30:15.364595 5284 tablet_service.cc:1514] Processing DeleteTablet for tablet b0bdf775e97f42b5ba4a04aedf23ae49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:15 UTC) from {username='slave'} at 127.0.0.1:51118
I20260619 12:30:15.364660 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet b0bdf775e97f42b5ba4a04aedf23ae49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:15 UTC) from {username='slave'} at 127.0.0.1:60430
I20260619 12:30:15.365244 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 5091
I20260619 12:30:15.365348 5521 tablet_replica.cc:330] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e: stopping tablet replica
I20260619 12:30:15.365661 5521 raft_consensus.cc:2229] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:30:15.366015 5521 raft_consensus.cc:2258] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:15.366395 5520 tablet_replica.cc:330] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc: stopping tablet replica
I20260619 12:30:15.366683 5520 raft_consensus.cc:2229] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:15.366987 5520 raft_consensus.cc:2258] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:15.368316 5521 ts_tablet_manager.cc:1905] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:15.368505 5520 ts_tablet_manager.cc:1905] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:15.370532 5521 ts_tablet_manager.cc:1918] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:30:15.370658 5521 log.cc:1196] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/b0bdf775e97f42b5ba4a04aedf23ae49
I20260619 12:30:15.370999 5521 ts_tablet_manager.cc:1939] T b0bdf775e97f42b5ba4a04aedf23ae49 P f2ae13a7e53446aebfa24760c86bf85e: Deleting consensus metadata
I20260619 12:30:15.371927 5520 ts_tablet_manager.cc:1918] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:30:15.372035 5520 log.cc:1196] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/b0bdf775e97f42b5ba4a04aedf23ae49
I20260619 12:30:15.372213 5018 catalog_manager.cc:4841] TS f2ae13a7e53446aebfa24760c86bf85e (127.0.98.3:46357): tablet b0bdf775e97f42b5ba4a04aedf23ae49 (table test-table [id=e6825f313fcf416eba754cf0e4018806]) successfully deleted
I20260619 12:30:15.372354 5520 ts_tablet_manager.cc:1939] T b0bdf775e97f42b5ba4a04aedf23ae49 P 5ccd9f737bfe481f9cc6a0c7b629ecfc: Deleting consensus metadata
I20260619 12:30:15.373596 5017 catalog_manager.cc:4841] TS 5ccd9f737bfe481f9cc6a0c7b629ecfc (127.0.98.2:36333): tablet b0bdf775e97f42b5ba4a04aedf23ae49 (table test-table [id=e6825f313fcf416eba754cf0e4018806]) successfully deleted
W20260619 12:30:15.374452 5017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40481 (error 108)
I20260619 12:30:15.374554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 5222
I20260619 12:30:15.375905 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:15.375634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40481 (local address 127.0.0.1:60442)
0619 12:30:15.375763 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:15.375834 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 12:30:15.376128 5017 catalog_manager.cc:4584] TS cfd6d0cf4da54309b260c810e1998063 (127.0.98.1:40481): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0bdf775e97f42b5ba4a04aedf23ae49: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40481: connect: Connection refused (error 111)
I20260619 12:30:15.382118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 5353
I20260619 12:30:15.388126 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 5000
2026-06-19T12:30:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1018 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 12:30:15.409725 392 mini_oidc.cc:152] Starting JWKS server
I20260619 12:30:15.411275 392 webserver.cc:460] Webserver started at https://127.0.0.1:35399/ using document root <none> and password file <none>
I20260619 12:30:15.411453 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 12:30:15.411731 392 webserver.cc:460] Webserver started at http://127.0.0.1:34593/ using document root <none> and password file <none>
2026-06-19T12:30:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:30:15Z Disabled control of system clock
I20260619 12:30:15.420379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40977
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40977
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35399/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0/testchainca.cert with env {}
W20260619 12:30:15.529240 5529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:15.529546 5529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:15.529628 5529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:15.533718 5529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:15.533814 5529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:15.533841 5529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:15.533882 5529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:15.533921 5529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 12:30:15.533977 5529 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 12:30:15.534017 5529 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35399/jwks.json
I20260619 12:30:15.539604 5529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40977
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35399/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:15.540686 5529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:15.541724 5529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:15.550227 5534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:15.550203 5535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:15.550203 5537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:15.550601 5529 server_base.cc:1034] running on GCE node
I20260619 12:30:15.551136 5529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:15.552198 5529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:15.553377 5529 hybrid_clock.cc:648] HybridClock initialized: now 1781872215553353 us; error 49 us; skew 500 ppm
I20260619 12:30:15.555379 5529 webserver.cc:460] Webserver started at http://127.0.98.62:41077/ using document root <none> and password file <none>
I20260619 12:30:15.555939 5529 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:15.556037 5529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:15.556254 5529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:15.558346 5529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "ebbba711fa0d4a97b4a0c880eb20d45f"
format_stamp: "Formatted at 2026-06-19 12:30:15 on dist-test-slave-fg3h"
server_key: "e546dff72c8a093640b82e6832e14bd6"
server_key_iv: "ad799cc28bc1d1ebf7f4ab9f4a87f0c7"
server_key_version: "encryptionkey@0"
I20260619 12:30:15.558928 5529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "ebbba711fa0d4a97b4a0c880eb20d45f"
format_stamp: "Formatted at 2026-06-19 12:30:15 on dist-test-slave-fg3h"
server_key: "e546dff72c8a093640b82e6832e14bd6"
server_key_iv: "ad799cc28bc1d1ebf7f4ab9f4a87f0c7"
server_key_version: "encryptionkey@0"
I20260619 12:30:15.568908 5529 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.000s
I20260619 12:30:15.572521 5543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:15.574226 5529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:30:15.574453 5529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "ebbba711fa0d4a97b4a0c880eb20d45f"
format_stamp: "Formatted at 2026-06-19 12:30:15 on dist-test-slave-fg3h"
server_key: "e546dff72c8a093640b82e6832e14bd6"
server_key_iv: "ad799cc28bc1d1ebf7f4ab9f4a87f0c7"
server_key_version: "encryptionkey@0"
I20260619 12:30:15.574618 5529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:15.591096 5529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:15.591848 5529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:15.592062 5529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:15.599305 5529 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40977
I20260619 12:30:15.599344 5595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40977 every 8 connection(s)
I20260619 12:30:15.600301 5529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:30:15.603600 5596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:15.609035 5596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f: Bootstrap starting.
I20260619 12:30:15.609711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 5529
I20260619 12:30:15.609840 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:30:15.610126 392 external_mini_cluster.cc:1428] Setting key cf6cf5dd06a0231c6a92044218cb61fc
I20260619 12:30:15.612032 5596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:15.612777 5596 log.cc:824] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:15.614704 5596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f: No bootstrap required, opened a new log
I20260619 12:30:15.618177 5596 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebbba711fa0d4a97b4a0c880eb20d45f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40977 } }
I20260619 12:30:15.618392 5596 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:15.618456 5596 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbba711fa0d4a97b4a0c880eb20d45f, State: Initialized, Role: FOLLOWER
I20260619 12:30:15.618937 5596 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebbba711fa0d4a97b4a0c880eb20d45f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40977 } }
I20260619 12:30:15.619097 5596 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:30:15.619182 5596 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:30:15.619284 5596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:15.620206 5596 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebbba711fa0d4a97b4a0c880eb20d45f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40977 } }
I20260619 12:30:15.620396 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:15.611480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38302 (local address 127.0.98.62:40977)
0619 12:30:15.612299 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:15.612316 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:15.612353 (+ 37us) server_negotiation.cc:407] Connection header received
0619 12:30:15.613171 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:15.613193 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:15.613556 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:15.613942 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:15.614420 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:15.615543 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:15.616559 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:15.617139 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:15.617336 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:15.618275 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:15.618284 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:15.618305 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:15.618809 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:15.618876 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:15.618899 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:15.619037 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:15.619268 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 12:30:15.619593 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":160,"threads_started":1}
I20260619 12:30:15.620626 5596 leader_election.cc:302] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebbba711fa0d4a97b4a0c880eb20d45f; no voters:
I20260619 12:30:15.620960 5596 leader_election.cc:288] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:30:15.621117 5601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:15.621381 5601 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f [term 1 LEADER]: Becoming Leader. State: Replica: ebbba711fa0d4a97b4a0c880eb20d45f, State: Running, Role: LEADER
I20260619 12:30:15.621711 5601 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebbba711fa0d4a97b4a0c880eb20d45f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40977 } }
I20260619 12:30:15.622246 5596 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:15.622953 5602 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebbba711fa0d4a97b4a0c880eb20d45f. Latest consensus state: current_term: 1 leader_uuid: "ebbba711fa0d4a97b4a0c880eb20d45f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebbba711fa0d4a97b4a0c880eb20d45f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40977 } } }
I20260619 12:30:15.623188 5602 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:15.623425 5603 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebbba711fa0d4a97b4a0c880eb20d45f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebbba711fa0d4a97b4a0c880eb20d45f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40977 } } }
I20260619 12:30:15.623529 5603 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:15.623628 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 12:30:15.627231 5616 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:30:15.627301 5616 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:30:15.627389 5610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:30:15.628144 5610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:30:15.634088 5610 catalog_manager.cc:1347] Generated new cluster ID: f6068f19a364462a8c8be45de8fc9202
I20260619 12:30:15.634198 5610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:30:15.649551 5610 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:30:15.650588 5610 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:30:15.666734 5610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebbba711fa0d4a97b4a0c880eb20d45f: Generated new TSK 0
I20260619 12:30:15.667562 5610 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 12:30:15.703656 5626 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 12:30:15.703811 5626 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 12:30:15.704386 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:15.686478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.0.98.62:40977)
0619 12:30:15.686619 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:15.686625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:15.686838 (+ 213us) server_negotiation.cc:407] Connection header received
0619 12:30:15.687040 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:15.687044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:15.687105 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:15.687228 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0619 12:30:15.687493 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:15.688350 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:15.689392 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:15.689747 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:15.689854 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:15.690185 (+ 331us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 12:30:15.703956 (+ 13771us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 12:30:15.704245 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38308: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 12:30:15.704469 5599 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38308: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 12:30:15.705647 5625 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:15.686371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40977 (local address 127.0.0.1:38308)
0619 12:30:15.686750 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:15.686765 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:15.686880 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:15.687255 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:15.687257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:15.687264 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 12:30:15.687374 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:15.687379 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:15.688473 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:15.688476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:15.689280 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:15.689290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:15.689849 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:15.689866 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:15.689942 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:15.689989 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 12:30:15.704207 (+ 14218us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 12:30:15.704402 (+ 195us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40977: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":123,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:40977: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 12:30:15.706527 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 5529
2026-06-19T12:30:15Z chronyd exiting
I20260619 12:30:15.743721 392 test_util.cc:175] -----------------------------------------------
I20260619 12:30:15.743808 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781872188319340-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (335 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 12:30:15.744796 392 mini_oidc.cc:152] Starting JWKS server
I20260619 12:30:15.745741 392 webserver.cc:460] Webserver started at https://127.0.0.1:45981/ using document root <none> and password file <none>
I20260619 12:30:15.745829 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 12:30:15.746033 392 webserver.cc:460] Webserver started at http://127.0.0.1:43031/ using document root <none> and password file <none>
2026-06-19T12:30:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:30:15Z Disabled control of system clock
I20260619 12:30:15.755064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37285
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37285
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45981/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872188319340-392-0/testchainca.cert with env {}
W20260619 12:30:15.869185 5632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:15.869450 5632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:15.869514 5632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:15.874683 5632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:15.874742 5632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:15.874758 5632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:15.874775 5632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:15.874790 5632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 12:30:15.874806 5632 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 12:30:15.874820 5632 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45981/jwks.json
I20260619 12:30:15.880489 5632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37285
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45981/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872188319340-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:15.881608 5632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:15.882562 5632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:30:15.891063 5632 server_base.cc:1034] running on GCE node
W20260619 12:30:15.891009 5637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:15.890987 5638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:15.891394 5640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:15.891907 5632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:15.893002 5632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:15.894184 5632 hybrid_clock.cc:648] HybridClock initialized: now 1781872215894167 us; error 28 us; skew 500 ppm
I20260619 12:30:15.896180 5632 webserver.cc:460] Webserver started at http://127.0.98.62:46547/ using document root <none> and password file <none>
I20260619 12:30:15.896807 5632 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:15.896874 5632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:15.897133 5632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:15.899250 5632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "b5462a22a64e42b1ace1545505937e7e"
format_stamp: "Formatted at 2026-06-19 12:30:15 on dist-test-slave-fg3h"
server_key: "3de31055f636205e1731a73c7172960c"
server_key_iv: "1d2d2e338a533377bd1f4d3b0cbc7996"
server_key_version: "encryptionkey@0"
I20260619 12:30:15.899835 5632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "b5462a22a64e42b1ace1545505937e7e"
format_stamp: "Formatted at 2026-06-19 12:30:15 on dist-test-slave-fg3h"
server_key: "3de31055f636205e1731a73c7172960c"
server_key_iv: "1d2d2e338a533377bd1f4d3b0cbc7996"
server_key_version: "encryptionkey@0"
I20260619 12:30:15.903545 5632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 12:30:15.905968 5646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:15.907330 5632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:15.907495 5632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "b5462a22a64e42b1ace1545505937e7e"
format_stamp: "Formatted at 2026-06-19 12:30:15 on dist-test-slave-fg3h"
server_key: "3de31055f636205e1731a73c7172960c"
server_key_iv: "1d2d2e338a533377bd1f4d3b0cbc7996"
server_key_version: "encryptionkey@0"
I20260619 12:30:15.907646 5632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:15.927099 5632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:15.927913 5632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:15.928148 5632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:15.935966 5632 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37285
I20260619 12:30:15.936024 5698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37285 every 8 connection(s)
I20260619 12:30:15.937078 5632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:30:15.940718 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:15.940903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 5632
I20260619 12:30:15.941138 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:30:15.941461 392 external_mini_cluster.cc:1428] Setting key 17c93a7fdc1c0a743d1b8d165b58bc26
I20260619 12:30:15.947162 5699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e: Bootstrap starting.
I20260619 12:30:15.949532 5699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:15.950373 5699 log.cc:824] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:15.950639 5702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:15.942785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50536 (local address 127.0.98.62:37285)
0619 12:30:15.943485 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:15.943507 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:15.943564 (+ 57us) server_negotiation.cc:407] Connection header received
0619 12:30:15.944236 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:15.944253 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:15.944660 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:15.945010 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:15.945459 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:15.946479 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:15.947370 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:15.947843 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:15.948013 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:15.948901 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:15.948907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:15.948919 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:15.949356 (+ 437us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:15.949409 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:15.949429 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:15.949546 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:15.949649 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 12:30:15.949896 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":161,"threads_started":1}
I20260619 12:30:15.952564 5699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e: No bootstrap required, opened a new log
I20260619 12:30:15.953882 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:30:15.956952 5699 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5462a22a64e42b1ace1545505937e7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37285 } }
I20260619 12:30:15.957234 5699 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:15.957309 5699 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5462a22a64e42b1ace1545505937e7e, State: Initialized, Role: FOLLOWER
I20260619 12:30:15.957787 5699 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5462a22a64e42b1ace1545505937e7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37285 } }
I20260619 12:30:15.957916 5699 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:30:15.957980 5699 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:30:15.958097 5699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:15.963469 5699 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5462a22a64e42b1ace1545505937e7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37285 } }
I20260619 12:30:15.963915 5699 leader_election.cc:302] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5462a22a64e42b1ace1545505937e7e; no voters:
I20260619 12:30:15.964329 5699 leader_election.cc:288] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:30:15.964532 5704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:15.964784 5704 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e [term 1 LEADER]: Becoming Leader. State: Replica: b5462a22a64e42b1ace1545505937e7e, State: Running, Role: LEADER
I20260619 12:30:15.965211 5704 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5462a22a64e42b1ace1545505937e7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37285 } }
I20260619 12:30:15.965951 5699 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:15.966892 5705 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5462a22a64e42b1ace1545505937e7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5462a22a64e42b1ace1545505937e7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37285 } } }
I20260619 12:30:15.967072 5705 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:15.967588 5706 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5462a22a64e42b1ace1545505937e7e. Latest consensus state: current_term: 1 leader_uuid: "b5462a22a64e42b1ace1545505937e7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5462a22a64e42b1ace1545505937e7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37285 } } }
I20260619 12:30:15.967681 5706 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:15.967834 5713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:30:15.971352 5713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:30:15.977176 5713 catalog_manager.cc:1347] Generated new cluster ID: 5c94269671b04b3692cb6c325a7fd14d
I20260619 12:30:15.977254 5713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:30:15.986522 5713 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:30:15.987466 5713 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:30:15.992440 5713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5462a22a64e42b1ace1545505937e7e: Generated new TSK 0
I20260619 12:30:15.993170 5713 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 12:30:16.028508 5729 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 12:30:16.028614 5729 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260619 12:30:16.029554 5728 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:16.007380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37285 (local address 127.0.0.1:50546)
0619 12:30:16.007681 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:16.007697 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:16.007826 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:16.008128 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:16.008132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:16.008140 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 12:30:16.008253 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:16.008260 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:16.009075 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:16.009079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:16.010122 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:16.010133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:16.010695 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:16.010718 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:16.010809 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:16.010837 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 12:30:16.029239 (+ 18402us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45981/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 12:30:16.029402 (+ 163us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37285: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45981/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":107,"threads_started":1}
I20260619 12:30:16.030190 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 5632
2026-06-19T12:30:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (324 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 12:30:16.069388 392 mini_oidc.cc:152] Starting JWKS server
I20260619 12:30:16.070271 392 webserver.cc:460] Webserver started at https://127.0.0.1:45355/ using document root <none> and password file <none>
I20260619 12:30:16.070384 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 12:30:16.070659 392 webserver.cc:460] Webserver started at http://127.0.0.1:43785/ using document root <none> and password file <none>
2026-06-19T12:30:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:30:16Z Disabled control of system clock
I20260619 12:30:16.080278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33757
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33757
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45355/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 12:30:16.197134 5735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:16.197424 5735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:16.197466 5735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:16.201758 5735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:16.201829 5735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:16.201845 5735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:16.201862 5735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:16.201877 5735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 12:30:16.201894 5735 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 12:30:16.201907 5735 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45355/jwks.json
I20260619 12:30:16.206992 5735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33757
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45355/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:16.208114 5735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:16.209548 5735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:16.218024 5740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:16.218014 5741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:16.218026 5743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:16.218900 5735 server_base.cc:1034] running on GCE node
I20260619 12:30:16.219491 5735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:16.220712 5735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:16.221967 5735 hybrid_clock.cc:648] HybridClock initialized: now 1781872216221951 us; error 30 us; skew 500 ppm
I20260619 12:30:16.224030 5735 webserver.cc:460] Webserver started at http://127.0.98.62:35405/ using document root <none> and password file <none>
I20260619 12:30:16.224625 5735 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:16.224733 5735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:16.224973 5735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:16.227170 5735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "d15fe92c660d42d7a8f93ac0957e796d"
format_stamp: "Formatted at 2026-06-19 12:30:16 on dist-test-slave-fg3h"
server_key: "969e8044cecced4b6f2e16e3cfe001b8"
server_key_iv: "703ea8c86b9adc86b782e9ca5858a345"
server_key_version: "encryptionkey@0"
I20260619 12:30:16.227866 5735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "d15fe92c660d42d7a8f93ac0957e796d"
format_stamp: "Formatted at 2026-06-19 12:30:16 on dist-test-slave-fg3h"
server_key: "969e8044cecced4b6f2e16e3cfe001b8"
server_key_iv: "703ea8c86b9adc86b782e9ca5858a345"
server_key_version: "encryptionkey@0"
I20260619 12:30:16.231611 5735 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 12:30:16.234217 5749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:16.235373 5735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:16.235539 5735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "d15fe92c660d42d7a8f93ac0957e796d"
format_stamp: "Formatted at 2026-06-19 12:30:16 on dist-test-slave-fg3h"
server_key: "969e8044cecced4b6f2e16e3cfe001b8"
server_key_iv: "703ea8c86b9adc86b782e9ca5858a345"
server_key_version: "encryptionkey@0"
I20260619 12:30:16.235684 5735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:16.247335 5735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:16.247929 5735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:16.248116 5735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:16.255046 5735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33757
I20260619 12:30:16.255065 5801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33757 every 8 connection(s)
I20260619 12:30:16.256011 5735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:30:16.259234 5802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:16.264712 5802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d: Bootstrap starting.
I20260619 12:30:16.266186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 5735
I20260619 12:30:16.266278 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:30:16.266523 392 external_mini_cluster.cc:1428] Setting key bcb4aa6ee4e6c76145043cc9e5ca2b92
I20260619 12:30:16.267539 5802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:16.268457 5802 log.cc:824] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:16.270829 5802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d: No bootstrap required, opened a new log
I20260619 12:30:16.275195 5802 raft_consensus.cc:348] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d15fe92c660d42d7a8f93ac0957e796d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33757 } }
I20260619 12:30:16.275491 5802 raft_consensus.cc:374] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:16.275563 5802 raft_consensus.cc:729] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d15fe92c660d42d7a8f93ac0957e796d, State: Initialized, Role: FOLLOWER
I20260619 12:30:16.276108 5802 consensus_queue.cc:260] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d15fe92c660d42d7a8f93ac0957e796d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33757 } }
I20260619 12:30:16.276283 5802 raft_consensus.cc:388] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:30:16.276340 5802 raft_consensus.cc:482] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:30:16.276438 5802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:16.277225 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:16.267624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37932 (local address 127.0.98.62:33757)
0619 12:30:16.268400 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:16.268419 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:16.268462 (+ 43us) server_negotiation.cc:407] Connection header received
0619 12:30:16.269436 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:16.269464 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:16.269893 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:16.270271 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:16.270783 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:16.272253 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:16.273226 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:16.273960 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:16.274168 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:16.274969 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:16.274975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:16.274991 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:16.275487 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:16.275564 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:16.275593 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:16.275705 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:16.275858 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 12:30:16.276132 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":156,"threads_started":1}
I20260619 12:30:16.277724 5802 raft_consensus.cc:504] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d15fe92c660d42d7a8f93ac0957e796d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33757 } }
I20260619 12:30:16.278090 5802 leader_election.cc:302] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d15fe92c660d42d7a8f93ac0957e796d; no voters:
I20260619 12:30:16.278439 5802 leader_election.cc:288] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:30:16.278561 5807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:16.278798 5807 raft_consensus.cc:686] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d [term 1 LEADER]: Becoming Leader. State: Replica: d15fe92c660d42d7a8f93ac0957e796d, State: Running, Role: LEADER
I20260619 12:30:16.279194 5807 consensus_queue.cc:237] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d15fe92c660d42d7a8f93ac0957e796d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33757 } }
I20260619 12:30:16.279829 5802 sys_catalog.cc:564] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:16.280536 5807 sys_catalog.cc:455] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d15fe92c660d42d7a8f93ac0957e796d. Latest consensus state: current_term: 1 leader_uuid: "d15fe92c660d42d7a8f93ac0957e796d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d15fe92c660d42d7a8f93ac0957e796d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33757 } } }
I20260619 12:30:16.280604 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:30:16.280668 5807 sys_catalog.cc:458] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:16.281036 5813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:30:16.281364 5808 sys_catalog.cc:455] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d15fe92c660d42d7a8f93ac0957e796d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d15fe92c660d42d7a8f93ac0957e796d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33757 } } }
I20260619 12:30:16.281472 5808 sys_catalog.cc:458] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:16.284569 5813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:30:16.290441 5813 catalog_manager.cc:1347] Generated new cluster ID: af9dc4f465c246d08271765cd85309ca
I20260619 12:30:16.290524 5813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:30:16.313234 5813 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:30:16.314509 5813 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:30:16.322744 5813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d15fe92c660d42d7a8f93ac0957e796d: Generated new TSK 0
I20260619 12:30:16.323598 5813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:30:16.339365 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:16.334104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.0.98.62:33757)
0619 12:30:16.334228 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:16.334233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:16.334383 (+ 150us) server_negotiation.cc:407] Connection header received
0619 12:30:16.334525 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:16.334529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:16.334585 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:16.334732 (+ 147us) server_negotiation.cc:226] Negotiated authn=JWT
0619 12:30:16.334897 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:16.335737 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:16.336724 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:16.337072 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:16.337200 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:16.337481 (+ 281us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 12:30:16.338958 (+ 1477us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45355/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 12:30:16.339193 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37940: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45355/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 12:30:16.339438 5805 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37940: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45355/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 12:30:16.341409 5831 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:16.334012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33757 (local address 127.0.0.1:37940)
0619 12:30:16.334278 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:16.334301 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:16.334399 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:16.334707 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:16.334709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:16.334715 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0619 12:30:16.334800 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:16.334804 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:16.335884 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:16.335888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:16.336590 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:16.336599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:16.337192 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:16.337214 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:16.337283 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:16.337306 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 12:30:16.339115 (+ 1809us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45355/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 12:30:16.341239 (+ 2124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33757: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45355/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":1840,"thread_start_us":102,"threads_started":1}
I20260619 12:30:16.341944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 5735
E20260619 12:30:16.349978 5832 webserver.cc:533] Webserver: sslize failed:
E20260619 12:30:16.350070 5832 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T12:30:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (312 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T12:30:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:30:16Z Disabled control of system clock
I20260619 12:30:16.390199 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36743
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36743
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872188319340-392-0/testchainca.cert with env {}
W20260619 12:30:16.500456 5836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:16.500741 5836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:16.500782 5836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:16.504819 5836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:16.504881 5836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:16.504897 5836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:16.504914 5836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:16.504930 5836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 12:30:16.504944 5836 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 12:30:16.504958 5836 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 12:30:16.510456 5836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872188319340-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:16.511488 5836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:16.512502 5836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:16.520923 5844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:16.520917 5842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:16.520917 5841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:16.521262 5836 server_base.cc:1034] running on GCE node
I20260619 12:30:16.521849 5836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:16.523007 5836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:16.524201 5836 hybrid_clock.cc:648] HybridClock initialized: now 1781872216524191 us; error 28 us; skew 500 ppm
I20260619 12:30:16.526110 5836 webserver.cc:460] Webserver started at http://127.0.98.62:42735/ using document root <none> and password file <none>
I20260619 12:30:16.526695 5836 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:16.526787 5836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:16.527009 5836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:16.528988 5836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "64f2749e3396458b8b4ebc4a58235e4b"
format_stamp: "Formatted at 2026-06-19 12:30:16 on dist-test-slave-fg3h"
server_key: "fcfc815fa1131bbd4958fe24fb656a88"
server_key_iv: "a19f9a6d46a4a547751cade312e7dad6"
server_key_version: "encryptionkey@0"
I20260619 12:30:16.529614 5836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "64f2749e3396458b8b4ebc4a58235e4b"
format_stamp: "Formatted at 2026-06-19 12:30:16 on dist-test-slave-fg3h"
server_key: "fcfc815fa1131bbd4958fe24fb656a88"
server_key_iv: "a19f9a6d46a4a547751cade312e7dad6"
server_key_version: "encryptionkey@0"
I20260619 12:30:16.533314 5836 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:30:16.535761 5850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:16.536979 5836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:30:16.537158 5836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "64f2749e3396458b8b4ebc4a58235e4b"
format_stamp: "Formatted at 2026-06-19 12:30:16 on dist-test-slave-fg3h"
server_key: "fcfc815fa1131bbd4958fe24fb656a88"
server_key_iv: "a19f9a6d46a4a547751cade312e7dad6"
server_key_version: "encryptionkey@0"
I20260619 12:30:16.537286 5836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:16.558930 5836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:16.559640 5836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:16.559850 5836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:16.566978 5836 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36743
I20260619 12:30:16.566998 5902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36743 every 8 connection(s)
I20260619 12:30:16.567935 5836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:30:16.571082 5903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:16.575958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 5836
I20260619 12:30:16.576084 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:30:16.576300 5903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b: Bootstrap starting.
I20260619 12:30:16.576395 392 external_mini_cluster.cc:1428] Setting key d6d6ab758b3931976372d40ed14f40a2
I20260619 12:30:16.578806 5903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:16.579540 5903 log.cc:824] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:16.581451 5903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b: No bootstrap required, opened a new log
I20260619 12:30:16.585426 5903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f2749e3396458b8b4ebc4a58235e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36743 } }
I20260619 12:30:16.585745 5903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:16.585827 5903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f2749e3396458b8b4ebc4a58235e4b, State: Initialized, Role: FOLLOWER
I20260619 12:30:16.586455 5903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f2749e3396458b8b4ebc4a58235e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36743 } }
I20260619 12:30:16.586580 5906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:16.577673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49332 (local address 127.0.98.62:36743)
0619 12:30:16.578370 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:16.578386 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:16.578425 (+ 39us) server_negotiation.cc:407] Connection header received
0619 12:30:16.579179 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:16.579198 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:16.579625 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:16.580039 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:16.580482 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:16.581679 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:16.582658 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:16.583230 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:16.583449 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:16.584486 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:16.584492 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:16.584506 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:16.584928 (+ 422us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:16.585018 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:16.585066 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:16.585234 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:16.585395 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 12:30:16.585672 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":123,"threads_started":1}
I20260619 12:30:16.586673 5903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:30:16.586756 5903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:30:16.586876 5903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:16.588006 5903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f2749e3396458b8b4ebc4a58235e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36743 } }
I20260619 12:30:16.588361 5903 leader_election.cc:302] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64f2749e3396458b8b4ebc4a58235e4b; no voters:
I20260619 12:30:16.588702 5903 leader_election.cc:288] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:30:16.588824 5908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:16.589049 5908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b [term 1 LEADER]: Becoming Leader. State: Replica: 64f2749e3396458b8b4ebc4a58235e4b, State: Running, Role: LEADER
I20260619 12:30:16.589510 5908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f2749e3396458b8b4ebc4a58235e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36743 } }
I20260619 12:30:16.589987 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:30:16.590194 5903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:16.591259 5909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64f2749e3396458b8b4ebc4a58235e4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f2749e3396458b8b4ebc4a58235e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36743 } } }
I20260619 12:30:16.591398 5909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:16.591734 5910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64f2749e3396458b8b4ebc4a58235e4b. Latest consensus state: current_term: 1 leader_uuid: "64f2749e3396458b8b4ebc4a58235e4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f2749e3396458b8b4ebc4a58235e4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36743 } } }
I20260619 12:30:16.591837 5910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:16.592206 5917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:30:16.595301 5917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:30:16.600482 5917 catalog_manager.cc:1347] Generated new cluster ID: 6f2175ccd7764faf86ba705067ac942e
I20260619 12:30:16.600554 5917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:30:16.613510 5917 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:30:16.614701 5917 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:30:16.633445 5917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64f2749e3396458b8b4ebc4a58235e4b: Generated new TSK 0
I20260619 12:30:16.634438 5917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:30:16.714329 5906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:16.648878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.0.98.62:36743)
0619 12:30:16.649012 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:16.649017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:16.649208 (+ 191us) server_negotiation.cc:407] Connection header received
0619 12:30:16.649345 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:16.649349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:16.649428 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:16.649546 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0619 12:30:16.649805 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:16.650631 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:16.651551 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:16.651902 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:16.652002 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:16.652326 (+ 324us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 12:30:16.713836 (+ 61510us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 12:30:16.714173 (+ 337us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49334: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 12:30:16.714424 5906 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49334: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 12:30:16.714586 5932 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:16.648786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36743 (local address 127.0.0.1:49334)
0619 12:30:16.649066 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:16.649124 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:16.649236 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:16.649592 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:16.649594 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:16.649602 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 12:30:16.649695 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:16.649700 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:16.650761 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:16.650764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:16.651440 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:16.651448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:16.652003 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:16.652024 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:16.652136 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:16.652158 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 12:30:16.714112 (+ 61954us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 12:30:16.714335 (+ 223us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36743: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":109,"threads_started":1}
I20260619 12:30:16.715284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 5836
2026-06-19T12:30:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (352 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (126 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (121 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:30:17 dist-test-slave-fg3h krb5kdc[5944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:30:17 dist-test-slave-fg3h krb5kdc[5944](info): set up 2 sockets
Jun 19 12:30:17 dist-test-slave-fg3h krb5kdc[5944](info): commencing operation
krb5kdc: starting...
W20260619 12:30:18.005909 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:30:18 dist-test-slave-fg3h krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:30:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:30:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:30:18.132097 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42547
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:18.242702 5957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:18.243031 5957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:18.243112 5957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:18.247274 5957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:18.247366 5957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:18.247401 5957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:18.247442 5957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:18.247474 5957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:30:18.252796 5957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:18.253945 5957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:18.254982 5957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:18.263787 5962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:18.263829 5963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:18.263811 5965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:18.264106 5957 server_base.cc:1034] running on GCE node
I20260619 12:30:18.264657 5957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:18.265873 5957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:18.267074 5957 hybrid_clock.cc:648] HybridClock initialized: now 1781872218267040 us; error 49 us; skew 500 ppm
Jun 19 12:30:18 dist-test-slave-fg3h krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:18.270056 5957 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:30:18.271132 5957 webserver.cc:460] Webserver started at http://127.0.98.62:37011/ using document root <none> and password file <none>
I20260619 12:30:18.271754 5957 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:18.271850 5957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:18.272091 5957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:18.274319 5957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "9b9727f7d59048828928eac34af0d64e"
format_stamp: "Formatted at 2026-06-19 12:30:18 on dist-test-slave-fg3h"
server_key: "cecdb9c3bbc5575958e86d8aefd2f85a"
server_key_iv: "0d5170e923c5b4cde6e8d1d9d2e4a680"
server_key_version: "encryptionkey@0"
I20260619 12:30:18.274911 5957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "9b9727f7d59048828928eac34af0d64e"
format_stamp: "Formatted at 2026-06-19 12:30:18 on dist-test-slave-fg3h"
server_key: "cecdb9c3bbc5575958e86d8aefd2f85a"
server_key_iv: "0d5170e923c5b4cde6e8d1d9d2e4a680"
server_key_version: "encryptionkey@0"
I20260619 12:30:18.278864 5957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:30:18.281420 5972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:18.282757 5957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 12:30:18.282898 5957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "9b9727f7d59048828928eac34af0d64e"
format_stamp: "Formatted at 2026-06-19 12:30:18 on dist-test-slave-fg3h"
server_key: "cecdb9c3bbc5575958e86d8aefd2f85a"
server_key_iv: "0d5170e923c5b4cde6e8d1d9d2e4a680"
server_key_version: "encryptionkey@0"
I20260619 12:30:18.283015 5957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:18.300483 5957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:18.303747 5957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:18.303964 5957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:18.311406 5957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42547
I20260619 12:30:18.311695 6024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42547 every 8 connection(s)
I20260619 12:30:18.312544 5957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:30:18.315526 6025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:18.318637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 5957
I20260619 12:30:18.318791 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:30:18.319110 392 external_mini_cluster.cc:1428] Setting key e4e793e991ef7d7372c247a0c5f8d270
I20260619 12:30:18.322412 6025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e: Bootstrap starting.
I20260619 12:30:18.325254 6025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:18.326180 6025 log.cc:824] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:18.328156 6025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e: No bootstrap required, opened a new log
Jun 19 12:30:18 dist-test-slave-fg3h krb5kdc[5944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:18.332453 6025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9727f7d59048828928eac34af0d64e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42547 } }
I20260619 12:30:18.332728 6025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:18.332780 6025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9727f7d59048828928eac34af0d64e, State: Initialized, Role: FOLLOWER
I20260619 12:30:18.333386 6025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9727f7d59048828928eac34af0d64e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42547 } }
I20260619 12:30:18.333552 6025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:30:18.333621 6025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:30:18.333714 6025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:18.334921 6025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9727f7d59048828928eac34af0d64e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42547 } }
I20260619 12:30:18.335336 6025 leader_election.cc:302] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b9727f7d59048828928eac34af0d64e; no voters:
I20260619 12:30:18.335651 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:18.320755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34962 (local address 127.0.98.62:42547)
0619 12:30:18.321283 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:18.321299 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:18.321334 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:30:18.321949 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:18.321966 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:18.322294 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:18.322624 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:18.323567 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:18.324588 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:18.325930 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:18.326486 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:18.326653 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:18.329372 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:18.329378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:18.329403 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:18.329436 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:18.332238 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:18.332746 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:18.332753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:18.332760 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:18.332934 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:18.333254 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:18.333257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:18.333259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:18.333818 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:18.334035 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:18.334239 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 12:30:18.334555 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":154,"threads_started":1}
I20260619 12:30:18.335688 6025 leader_election.cc:288] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:30:18.336333 6030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:18.336553 6030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e [term 1 LEADER]: Becoming Leader. State: Replica: 9b9727f7d59048828928eac34af0d64e, State: Running, Role: LEADER
I20260619 12:30:18.336848 6030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9727f7d59048828928eac34af0d64e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42547 } }
I20260619 12:30:18.336974 6025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:18.338125 6031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b9727f7d59048828928eac34af0d64e. Latest consensus state: current_term: 1 leader_uuid: "9b9727f7d59048828928eac34af0d64e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9727f7d59048828928eac34af0d64e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42547 } } }
I20260619 12:30:18.338255 6031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:18.338605 6032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b9727f7d59048828928eac34af0d64e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9727f7d59048828928eac34af0d64e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42547 } } }
I20260619 12:30:18.338691 6039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:30:18.338697 6032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:18.341867 6039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:30:18.347506 6039 catalog_manager.cc:1347] Generated new cluster ID: e1b0159d897747358947807caeecf485
I20260619 12:30:18.347587 6039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:30:18.379082 6039 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:30:18.380404 6039 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:30:18.388442 6039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b9727f7d59048828928eac34af0d64e: Generated new TSK 0
I20260619 12:30:18.389184 6039 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:30:18.424150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42547
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:18.536571 6051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:18.536880 6051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:18.536962 6051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:18.541429 6051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:18.541509 6051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:18.541602 6051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:30:18.546797 6051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:18.547859 6051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:18.548883 6051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:18.559031 6057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:18.559067 6059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:18.559039 6056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:18.559157 6051 server_base.cc:1034] running on GCE node
I20260619 12:30:18.559772 6051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:18.560366 6051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:18.561546 6051 hybrid_clock.cc:648] HybridClock initialized: now 1781872218561528 us; error 31 us; skew 500 ppm
Jun 19 12:30:18 dist-test-slave-fg3h krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:18.568475 6051 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:30:18.569396 6051 webserver.cc:460] Webserver started at http://127.0.98.1:42363/ using document root <none> and password file <none>
I20260619 12:30:18.569897 6051 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:18.569958 6051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:18.570114 6051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:18.572022 6051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "660802a60b464ee79eb9d6bda25d00cb"
format_stamp: "Formatted at 2026-06-19 12:30:18 on dist-test-slave-fg3h"
server_key: "dc28eb88bb20ca3ccb9b62f7b544eb63"
server_key_iv: "2a47059acbaf2ae706ced4cb6208ba5c"
server_key_version: "encryptionkey@0"
I20260619 12:30:18.572506 6051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "660802a60b464ee79eb9d6bda25d00cb"
format_stamp: "Formatted at 2026-06-19 12:30:18 on dist-test-slave-fg3h"
server_key: "dc28eb88bb20ca3ccb9b62f7b544eb63"
server_key_iv: "2a47059acbaf2ae706ced4cb6208ba5c"
server_key_version: "encryptionkey@0"
I20260619 12:30:18.576078 6051 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 12:30:18.578488 6066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:18.579669 6051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:30:18.579790 6051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "660802a60b464ee79eb9d6bda25d00cb"
format_stamp: "Formatted at 2026-06-19 12:30:18 on dist-test-slave-fg3h"
server_key: "dc28eb88bb20ca3ccb9b62f7b544eb63"
server_key_iv: "2a47059acbaf2ae706ced4cb6208ba5c"
server_key_version: "encryptionkey@0"
I20260619 12:30:18.579892 6051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:18.612995 6051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:18.616401 6051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:18.616607 6051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:18.617270 6051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:18.618077 6051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:18.618124 6051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:18.618184 6051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:18.618216 6051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:18.627301 6051 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43855
I20260619 12:30:18.627326 6179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43855 every 8 connection(s)
I20260619 12:30:18.628183 6051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:30:18.631373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 6051
I20260619 12:30:18.631484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:30:18.631747 392 external_mini_cluster.cc:1428] Setting key f602c1a2910ae016e1b148dd9f6ec149
Jun 19 12:30:18 dist-test-slave-fg3h krb5kdc[5944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:30:18.644560 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:18.630143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41005 (local address 127.0.98.62:42547)
0619 12:30:18.630281 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:18.630285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:18.631123 (+ 838us) server_negotiation.cc:407] Connection header received
0619 12:30:18.632904 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:18.632907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:18.632953 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:18.633057 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:18.634380 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:18.635065 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:18.636766 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:18.637220 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:18.637352 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:18.640348 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:18.640363 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:18.640366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:18.640413 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:18.642306 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:18.642891 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:18.642894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:18.642896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:18.642984 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:18.643432 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:18.643436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:18.643438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:18.643629 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:18.643712 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:18.644253 (+ 541us) server_negotiation.cc:299] Negotiation successful
0619 12:30:18.644432 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:30:18.645509 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:18.630509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42547 (local address 127.0.98.1:41005)
0619 12:30:18.630951 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:18.630994 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:18.632677 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:18.633331 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:18.633343 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:18.633720 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:18.634224 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:18.634239 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:18.635200 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:18.635207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:18.636646 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:18.636657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:18.637378 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:18.637381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:18.637700 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:18.638346 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:18.638376 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:18.640160 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:18.642482 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:18.642496 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:18.642508 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:18.642766 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:18.643088 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:18.643090 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:18.643092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:18.643226 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:18.643728 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:18.643733 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:18.643972 (+ 239us) client_negotiation.cc:769] Sending connection context
0619 12:30:18.644136 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 12:30:18.644490 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":99,"threads_started":1}
I20260619 12:30:18.646986 6180 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42547
I20260619 12:30:18.647260 6180 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:18.647760 6180 heartbeater.cc:511] Master 127.0.98.62:42547 requested a full tablet report, sending...
I20260619 12:30:18.649550 5989 ts_manager.cc:194] Registered new tserver with Master: 660802a60b464ee79eb9d6bda25d00cb (127.0.98.1:43855)
I20260619 12:30:18.651436 5989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:30:18.664117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42547
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:18.774056 6185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:18.774319 6185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:18.774405 6185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:18.778489 6185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:18.778551 6185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:18.778649 6185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:30:18.784201 6185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:18.785347 6185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:18.786381 6185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:18.796468 6193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:18.796622 6191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:18.796679 6185 server_base.cc:1034] running on GCE node
W20260619 12:30:18.796629 6190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:18.797335 6185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:18.798045 6185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:18.799213 6185 hybrid_clock.cc:648] HybridClock initialized: now 1781872218799184 us; error 47 us; skew 500 ppm
Jun 19 12:30:18 dist-test-slave-fg3h krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:18.802539 6185 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:30:18.803366 6185 webserver.cc:460] Webserver started at http://127.0.98.2:36221/ using document root <none> and password file <none>
I20260619 12:30:18.803889 6185 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:18.803944 6185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:18.804109 6185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:18.806304 6185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "0695ac96e76143619ce30cff1fb7c6c2"
format_stamp: "Formatted at 2026-06-19 12:30:18 on dist-test-slave-fg3h"
server_key: "63c5ca61145c9bd735dd40add3d25fbc"
server_key_iv: "f7288ec254fce76babca20a45cd054df"
server_key_version: "encryptionkey@0"
I20260619 12:30:18.806835 6185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0695ac96e76143619ce30cff1fb7c6c2"
format_stamp: "Formatted at 2026-06-19 12:30:18 on dist-test-slave-fg3h"
server_key: "63c5ca61145c9bd735dd40add3d25fbc"
server_key_iv: "f7288ec254fce76babca20a45cd054df"
server_key_version: "encryptionkey@0"
I20260619 12:30:18.810652 6185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:30:18.813323 6200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:18.814601 6185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:30:18.814741 6185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "0695ac96e76143619ce30cff1fb7c6c2"
format_stamp: "Formatted at 2026-06-19 12:30:18 on dist-test-slave-fg3h"
server_key: "63c5ca61145c9bd735dd40add3d25fbc"
server_key_iv: "f7288ec254fce76babca20a45cd054df"
server_key_version: "encryptionkey@0"
I20260619 12:30:18.814864 6185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:18.828558 6185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:18.832149 6185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:18.832365 6185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:18.832990 6185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:18.833865 6185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:18.833945 6185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:18.834019 6185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:18.834070 6185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:18.844046 6185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37901
I20260619 12:30:18.844092 6313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37901 every 8 connection(s)
I20260619 12:30:18.845077 6185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:30:18.850773 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 6185
I20260619 12:30:18.850910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:30:18.851193 392 external_mini_cluster.cc:1428] Setting key 49efe04b3e76b1fd1ff76a87f9f87596
Jun 19 12:30:18 dist-test-slave-fg3h krb5kdc[5944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:30:18.861478 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:18.847424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47523 (local address 127.0.98.62:42547)
0619 12:30:18.847607 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:18.847621 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:18.848435 (+ 814us) server_negotiation.cc:407] Connection header received
0619 12:30:18.849892 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:18.849895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:18.849956 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:18.850043 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:18.851566 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:18.852251 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:18.854146 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:18.854386 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:18.854506 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:18.857222 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:18.857226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:18.857229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:18.857263 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:18.859066 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:18.859666 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:18.859670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:18.859672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:18.859768 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:18.860152 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:18.860156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:18.860158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:18.860414 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:18.860544 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:18.861117 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 12:30:18.861261 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 12:30:18.862346 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:18.847781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42547 (local address 127.0.98.2:47523)
0619 12:30:18.848262 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:18.848307 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:18.849675 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:18.850276 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:18.850283 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:18.850736 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:18.851402 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:18.851417 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:18.852639 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:18.852644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:18.854004 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:18.854014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:18.854547 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:18.854551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:18.854724 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:18.855426 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:18.855443 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:18.856965 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:18.859204 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:18.859210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:18.859225 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:18.859535 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:18.859886 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:18.859888 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:18.859890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:18.860035 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:18.860535 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:18.860541 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:18.860795 (+ 254us) client_negotiation.cc:769] Sending connection context
0619 12:30:18.861031 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 12:30:18.861437 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":118,"threads_started":1}
I20260619 12:30:18.863629 6314 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42547
I20260619 12:30:18.863853 6314 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 12:30:18.864408 6314 heartbeater.cc:511] Master 127.0.98.62:42547 requested a full tablet report, sending...
I20260619 12:30:18.865569 5989 ts_manager.cc:194] Registered new tserver with Master: 0695ac96e76143619ce30cff1fb7c6c2 (127.0.98.2:37901)
I20260619 12:30:18.866400 5989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:30:18.880744 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42547
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:18.992372 6319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:18.992672 6319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:18.992748 6319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:18.996867 6319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:18.996963 6319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:18.997058 6319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:30:19.002393 6319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:19.003470 6319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:19.004503 6319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:19.014191 6324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:19.014178 6327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:19.014178 6325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:19.014746 6319 server_base.cc:1034] running on GCE node
I20260619 12:30:19.015229 6319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:19.015888 6319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:19.017122 6319 hybrid_clock.cc:648] HybridClock initialized: now 1781872219017072 us; error 37 us; skew 500 ppm
Jun 19 12:30:19 dist-test-slave-fg3h krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:19.020149 6319 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:30:19.021178 6319 webserver.cc:460] Webserver started at http://127.0.98.3:37499/ using document root <none> and password file <none>
I20260619 12:30:19.021783 6319 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:19.021878 6319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:19.022115 6319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:19.024170 6319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "0a027123cdef486eb9c5a654771f77a4"
format_stamp: "Formatted at 2026-06-19 12:30:19 on dist-test-slave-fg3h"
server_key: "192fbec9746e9921b1251785a870dfc6"
server_key_iv: "581fb7226ea9ad726a3100f9998d43b6"
server_key_version: "encryptionkey@0"
I20260619 12:30:19.024773 6319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0a027123cdef486eb9c5a654771f77a4"
format_stamp: "Formatted at 2026-06-19 12:30:19 on dist-test-slave-fg3h"
server_key: "192fbec9746e9921b1251785a870dfc6"
server_key_iv: "581fb7226ea9ad726a3100f9998d43b6"
server_key_version: "encryptionkey@0"
I20260619 12:30:19.028652 6319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:30:19.031255 6334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:19.032536 6319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:30:19.032675 6319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "0a027123cdef486eb9c5a654771f77a4"
format_stamp: "Formatted at 2026-06-19 12:30:19 on dist-test-slave-fg3h"
server_key: "192fbec9746e9921b1251785a870dfc6"
server_key_iv: "581fb7226ea9ad726a3100f9998d43b6"
server_key_version: "encryptionkey@0"
I20260619 12:30:19.032804 6319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:19.065299 6319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:19.068418 6319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:19.068657 6319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:19.069304 6319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:19.070137 6319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:19.070209 6319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:19.070278 6319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:19.070328 6319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:19.080380 6319 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38251
I20260619 12:30:19.080410 6447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38251 every 8 connection(s)
I20260619 12:30:19.081367 6319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:30:19.087677 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 6319
I20260619 12:30:19.087817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:30:19.088136 392 external_mini_cluster.cc:1428] Setting key 330594e35e44b30b9b0f3daf825af5ec
Jun 19 12:30:19 dist-test-slave-fg3h krb5kdc[5944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:19.096151 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:19.083232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59083 (local address 127.0.98.62:42547)
0619 12:30:19.083415 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:19.083420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:19.084355 (+ 935us) server_negotiation.cc:407] Connection header received
0619 12:30:19.085837 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:19.085840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:19.085888 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:19.085970 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:19.087243 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:19.088025 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:19.089423 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:19.089724 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:19.089839 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:19.092517 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:19.092521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:19.092523 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:19.092548 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:19.094141 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:19.094687 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:19.094690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:19.094691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:19.094769 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:19.095111 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:19.095115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:19.095116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:19.095303 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:19.095390 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:19.095920 (+ 530us) server_negotiation.cc:299] Negotiation successful
0619 12:30:19.096028 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 12:30:19.096992 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:19.083750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42547 (local address 127.0.98.3:59083)
0619 12:30:19.084199 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:19.084242 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:19.085633 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:19.086128 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:19.086135 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:19.086608 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:19.087091 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:19.087104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:19.088157 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:19.088161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:19.089306 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:19.089316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:19.089909 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:19.089913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:19.090111 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:19.090771 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:19.090788 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:19.092290 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:19.094256 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:19.094261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:19.094275 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:19.094591 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:19.094851 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:19.094853 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:19.094854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:19.094989 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:19.095402 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:19.095407 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:19.095680 (+ 273us) client_negotiation.cc:769] Sending connection context
0619 12:30:19.095845 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 12:30:19.096140 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":107,"threads_started":1}
I20260619 12:30:19.098136 6448 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42547
I20260619 12:30:19.098402 6448 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:19.098928 6448 heartbeater.cc:511] Master 127.0.98.62:42547 requested a full tablet report, sending...
I20260619 12:30:19.100039 5989 ts_manager.cc:194] Registered new tserver with Master: 0a027123cdef486eb9c5a654771f77a4 (127.0.98.3:38251)
I20260619 12:30:19.100669 5989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59083
I20260619 12:30:19.102736 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/test-admin.keytab.
Jun 19 12:30:19 dist-test-slave-fg3h krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:19.118396 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 12:30:19.121845 6455 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:19.120165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39732 (local address 127.0.98.1:43855)
0619 12:30:19.120541 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:19.120558 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:19.120581 (+ 23us) server_negotiation.cc:407] Connection header received
0619 12:30:19.120630 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:19.120634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:19.120819 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:19.120947 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:19.121654 (+ 707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39732: BlockingRecv error: recv got EOF from 127.0.0.1:39732 (error 108)
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
W20260619 12:30:19.121949 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 12:30:19 dist-test-slave-fg3h krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:19.123122 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 12:30:19 dist-test-slave-fg3h krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:19.124223 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 12:30:19 dist-test-slave-fg3h krb5kdc[5944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:30:19.134337 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:19.125373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39740 (local address 127.0.98.1:43855)
0619 12:30:19.125502 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:19.125506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:19.125655 (+ 149us) server_negotiation.cc:407] Connection header received
0619 12:30:19.125790 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:19.125793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:19.125836 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:19.125916 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:19.126484 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:19.127252 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:19.128067 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:19.128468 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:19.128554 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:19.130912 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:19.130921 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:19.130934 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:19.130956 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:19.132665 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:19.133158 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:19.133164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:19.133165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:19.133251 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:19.133500 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:19.133502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:19.133504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:19.133745 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:19.133909 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:19.134079 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 12:30:19.134200 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 12:30:19.138335 6455 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:19.137395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39744 (local address 127.0.98.1:43855)
0619 12:30:19.137549 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:19.137553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:19.137566 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:30:19.137666 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:19.137669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:19.137712 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:19.137821 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:19.138228 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39744: BlockingRecv error: recv got EOF from 127.0.0.1:39744 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 12:30:19.138417 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 12:30:19 dist-test-slave-fg3h krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:19.139457 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 12:30:19 dist-test-slave-fg3h krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:19.140298 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 12:30:19 dist-test-slave-fg3h krb5kdc[5944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:30:19.148998 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:19.141032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39746 (local address 127.0.98.1:43855)
0619 12:30:19.141171 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:19.141175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:19.141309 (+ 134us) server_negotiation.cc:407] Connection header received
0619 12:30:19.141481 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:19.141484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:19.141521 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:19.141610 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:19.142081 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:19.142596 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:19.143381 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:19.143615 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:19.143723 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:19.145966 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:19.145970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:19.145973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:19.145996 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:19.147382 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:19.147826 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:19.147830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:19.147832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:19.147920 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:19.148186 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:19.148191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:19.148193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:19.148357 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:19.148490 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:19.148715 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 12:30:19.148818 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 12:30:19.151635 6455 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:19.150614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39760 (local address 127.0.98.1:43855)
0619 12:30:19.150720 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:19.150724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:19.150859 (+ 135us) server_negotiation.cc:407] Connection header received
0619 12:30:19.150944 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:19.150946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:19.150978 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:19.151041 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:19.151533 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39760: BlockingRecv error: recv got EOF from 127.0.0.1:39760 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 12:30:19.151726 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781872188319340-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 12:30:19 dist-test-slave-fg3h krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:19.152653 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 12:30:19 dist-test-slave-fg3h krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:19.153640 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 12:30:19 dist-test-slave-fg3h krb5kdc[5944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:30:19.162279 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:19.154404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39762 (local address 127.0.98.1:43855)
0619 12:30:19.154538 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:19.154542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:19.154664 (+ 122us) server_negotiation.cc:407] Connection header received
0619 12:30:19.154796 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:19.154800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:19.154838 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:19.154929 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:19.155396 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:19.155954 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:19.156748 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:19.156973 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:19.157054 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:19.159365 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:19.159368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:19.159370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:19.159440 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:19.160885 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:19.161321 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:19.161325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:19.161326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:19.161398 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:19.161620 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:19.161622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:19.161623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:19.161758 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:19.161847 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:19.162082 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 12:30:19.162167 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 12:30:19.163333 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 6051
I20260619 12:30:19.169006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 6185
I20260619 12:30:19.175390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 6319
I20260619 12:30:19.180457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 5957
2026-06-19T12:30:19Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2219 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:30:19 dist-test-slave-fg3h krb5kdc[6468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:30:19 dist-test-slave-fg3h krb5kdc[6468](info): set up 2 sockets
Jun 19 12:30:19 dist-test-slave-fg3h krb5kdc[6468](info): commencing operation
krb5kdc: starting...
W20260619 12:30:20.337566 392 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:30:20 dist-test-slave-fg3h krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:30:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:30:20Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 12:30:20.460227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43543
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:20.571789 6481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:20.572044 6481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:20.572084 6481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:20.576159 6481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:20.576218 6481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:20.576236 6481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:20.576253 6481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:20.576268 6481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:30:20.581578 6481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:20.582667 6481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:20.583699 6481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:20.592340 6487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:20.592325 6486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:20.592311 6489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:20.592463 6481 server_base.cc:1034] running on GCE node
I20260619 12:30:20.593225 6481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:20.594386 6481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:20.595616 6481 hybrid_clock.cc:648] HybridClock initialized: now 1781872220595582 us; error 48 us; skew 500 ppm
Jun 19 12:30:20 dist-test-slave-fg3h krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:20.598435 6481 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 12:30:20.599368 6481 webserver.cc:460] Webserver started at http://127.0.98.62:39169/ using document root <none> and password file <none>
I20260619 12:30:20.599912 6481 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:20.599972 6481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:20.600136 6481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:20.602217 6481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "5163b08470e840a98fb193929dc7ad25"
format_stamp: "Formatted at 2026-06-19 12:30:20 on dist-test-slave-fg3h"
server_key: "8996d089a9258f55820b20a2dc2bc209"
server_key_iv: "b87b5f0df0276c681e6f0c2b34d1d2d4"
server_key_version: "encryptionkey@0"
I20260619 12:30:20.602727 6481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "5163b08470e840a98fb193929dc7ad25"
format_stamp: "Formatted at 2026-06-19 12:30:20 on dist-test-slave-fg3h"
server_key: "8996d089a9258f55820b20a2dc2bc209"
server_key_iv: "b87b5f0df0276c681e6f0c2b34d1d2d4"
server_key_version: "encryptionkey@0"
I20260619 12:30:20.606349 6481 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:30:20.608702 6496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:20.609963 6481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:20.610065 6481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "5163b08470e840a98fb193929dc7ad25"
format_stamp: "Formatted at 2026-06-19 12:30:20 on dist-test-slave-fg3h"
server_key: "8996d089a9258f55820b20a2dc2bc209"
server_key_iv: "b87b5f0df0276c681e6f0c2b34d1d2d4"
server_key_version: "encryptionkey@0"
I20260619 12:30:20.610149 6481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:20.660187 6481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:20.663344 6481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:20.663514 6481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:20.670765 6481 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43543
I20260619 12:30:20.671066 6548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43543 every 8 connection(s)
I20260619 12:30:20.671756 6481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:30:20.674489 6549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:20.677457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 6481
I20260619 12:30:20.677590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:30:20.677874 392 external_mini_cluster.cc:1428] Setting key a3bcfaa3830fa57fa8210a88f601e823
I20260619 12:30:20.680441 6549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25: Bootstrap starting.
I20260619 12:30:20.682947 6549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:20.683832 6549 log.cc:824] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:20.686158 6549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25: No bootstrap required, opened a new log
Jun 19 12:30:20 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:30:20.689443 6549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } }
I20260619 12:30:20.689687 6549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:20.689770 6549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5163b08470e840a98fb193929dc7ad25, State: Initialized, Role: FOLLOWER
I20260619 12:30:20.690237 6549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } }
I20260619 12:30:20.690384 6549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:30:20.690474 6549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:30:20.690574 6549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:20.691438 6549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } }
I20260619 12:30:20.691776 6549 leader_election.cc:302] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5163b08470e840a98fb193929dc7ad25; no voters:
I20260619 12:30:20.692154 6549 leader_election.cc:288] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:30:20.692209 6554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:20.692384 6554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [term 1 LEADER]: Becoming Leader. State: Replica: 5163b08470e840a98fb193929dc7ad25, State: Running, Role: LEADER
I20260619 12:30:20.692756 6554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } }
I20260619 12:30:20.693001 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:20.679224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35650 (local address 127.0.98.62:43543)
0619 12:30:20.679618 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:20.679638 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:20.679688 (+ 50us) server_negotiation.cc:407] Connection header received
0619 12:30:20.680316 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:20.680333 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:20.680707 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:20.680982 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:20.681917 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:20.682932 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:20.684069 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:20.684546 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:20.684680 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:20.687500 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:20.687506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:20.687518 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:20.687544 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:20.689907 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:20.690337 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:20.690342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:20.690348 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:20.690545 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:20.690843 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:20.690847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:20.690848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:20.691243 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:20.691450 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:20.691731 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 12:30:20.692019 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":145,"threads_started":1}
I20260619 12:30:20.693396 6549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:20.694563 6555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5163b08470e840a98fb193929dc7ad25. Latest consensus state: current_term: 1 leader_uuid: "5163b08470e840a98fb193929dc7ad25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } }
I20260619 12:30:20.694722 6555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:20.695076 6556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5163b08470e840a98fb193929dc7ad25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } }
I20260619 12:30:20.695173 6556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:20.695742 6563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:30:20.698575 6563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:30:20.704008 6563 catalog_manager.cc:1347] Generated new cluster ID: 1e3227015b29436ab0201074af2dcaf3
I20260619 12:30:20.704097 6563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:30:20.735782 6563 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:30:20.737133 6563 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:30:20.746935 6563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25: Generated new TSK 0
I20260619 12:30:20.747794 6563 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 12:30:20.784541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43543
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:20.898514 6575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:20.898772 6575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:20.898855 6575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:20.903065 6575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:20.903132 6575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:20.903241 6575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:30:20.908453 6575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:20.909469 6575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:20.910511 6575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:20.920495 6580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:20.920481 6581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:20.920483 6583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:20.920804 6575 server_base.cc:1034] running on GCE node
I20260619 12:30:20.921351 6575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:20.921978 6575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:20.923134 6575 hybrid_clock.cc:648] HybridClock initialized: now 1781872220923117 us; error 29 us; skew 500 ppm
Jun 19 12:30:20 dist-test-slave-fg3h krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:20.925837 6575 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 12:30:20.926625 6575 webserver.cc:460] Webserver started at http://127.0.98.1:37057/ using document root <none> and password file <none>
I20260619 12:30:20.927135 6575 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:20.927191 6575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:20.927350 6575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:20.929380 6575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "79139bcb3a2c449996505234878d5ed2"
format_stamp: "Formatted at 2026-06-19 12:30:20 on dist-test-slave-fg3h"
server_key: "932e32ed5860f203555e0004fa4035c1"
server_key_iv: "1cc52a7873fadfb353b9824dff81cd4d"
server_key_version: "encryptionkey@0"
I20260619 12:30:20.929878 6575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "79139bcb3a2c449996505234878d5ed2"
format_stamp: "Formatted at 2026-06-19 12:30:20 on dist-test-slave-fg3h"
server_key: "932e32ed5860f203555e0004fa4035c1"
server_key_iv: "1cc52a7873fadfb353b9824dff81cd4d"
server_key_version: "encryptionkey@0"
I20260619 12:30:20.933350 6575 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:30:20.935616 6590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:20.936808 6575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:30:20.936929 6575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "79139bcb3a2c449996505234878d5ed2"
format_stamp: "Formatted at 2026-06-19 12:30:20 on dist-test-slave-fg3h"
server_key: "932e32ed5860f203555e0004fa4035c1"
server_key_iv: "1cc52a7873fadfb353b9824dff81cd4d"
server_key_version: "encryptionkey@0"
I20260619 12:30:20.937031 6575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:20.949728 6575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:20.952575 6575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:20.952768 6575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:20.953380 6575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:20.954147 6575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:20.954197 6575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:20.954258 6575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:20.954290 6575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:20.962813 6575 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42353
I20260619 12:30:20.962838 6703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42353 every 8 connection(s)
I20260619 12:30:20.963640 6575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:30:20.971190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 6575
I20260619 12:30:20.971331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:30:20.971683 392 external_mini_cluster.cc:1428] Setting key b90418c7724ad8297f742a2ed06a1feb
Jun 19 12:30:20 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:30:20.978133 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:20.965707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59589 (local address 127.0.98.62:43543)
0619 12:30:20.965839 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:20.965843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:20.966711 (+ 868us) server_negotiation.cc:407] Connection header received
0619 12:30:20.967977 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:20.967980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:20.968025 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:20.968101 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:20.969310 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:20.969897 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:20.971042 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:20.971465 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:20.971685 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:20.974315 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:20.974321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:20.974328 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:20.974363 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:20.975816 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:20.976414 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:20.976416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:20.976417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:20.976486 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:20.976912 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:20.976914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:20.976915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:20.977128 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:20.977215 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:20.977909 (+ 694us) server_negotiation.cc:299] Negotiation successful
0619 12:30:20.978014 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:30:20.979434 6706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:20.966052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.1:59589)
0619 12:30:20.966504 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:20.966558 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:20.967776 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:20.968257 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:20.968264 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:20.968656 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:20.969170 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:20.969183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:20.970016 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:20.970018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:20.970903 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:20.970912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:20.971699 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:20.971702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:20.971915 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:20.972548 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:20.972565 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:20.974154 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:20.975932 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:20.975938 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:20.975956 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:20.976321 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:20.976574 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:20.976580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:20.976585 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:20.976816 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:20.977239 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:20.977249 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:20.977600 (+ 351us) client_negotiation.cc:769] Sending connection context
0619 12:30:20.977848 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 12:30:20.978239 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":108,"threads_started":1}
I20260619 12:30:20.980697 6704 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43543
I20260619 12:30:20.980943 6704 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:20.981472 6704 heartbeater.cc:511] Master 127.0.98.62:43543 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:30:20.983047 6513 ts_manager.cc:194] Registered new tserver with Master: 79139bcb3a2c449996505234878d5ed2 (127.0.98.1:42353)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260619 12:30:20.984278 6513 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 12:30:21.000317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43543
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:21.114187 6709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:21.114446 6709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:21.114490 6709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:21.118745 6709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:21.118808 6709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:21.118898 6709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:30:21.124109 6709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:21.125206 6709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:21.126248 6709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:21.136406 6717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:21.136332 6715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:21.136370 6714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:21.136592 6709 server_base.cc:1034] running on GCE node
I20260619 12:30:21.137146 6709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:21.137795 6709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:21.138980 6709 hybrid_clock.cc:648] HybridClock initialized: now 1781872221138966 us; error 30 us; skew 500 ppm
Jun 19 12:30:21 dist-test-slave-fg3h krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:21.142299 6709 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 12:30:21.143244 6709 webserver.cc:460] Webserver started at http://127.0.98.2:45557/ using document root <none> and password file <none>
I20260619 12:30:21.143776 6709 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:21.143837 6709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:21.143998 6709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:21.146028 6709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "d437cfac39e64c909043a1c57af6668f"
format_stamp: "Formatted at 2026-06-19 12:30:21 on dist-test-slave-fg3h"
server_key: "d5716c29bd498381d019d2586ec08739"
server_key_iv: "dcc607c78a875be32877c99b4ee57708"
server_key_version: "encryptionkey@0"
I20260619 12:30:21.146615 6709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d437cfac39e64c909043a1c57af6668f"
format_stamp: "Formatted at 2026-06-19 12:30:21 on dist-test-slave-fg3h"
server_key: "d5716c29bd498381d019d2586ec08739"
server_key_iv: "dcc607c78a875be32877c99b4ee57708"
server_key_version: "encryptionkey@0"
I20260619 12:30:21.150254 6709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 12:30:21.152586 6724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:21.153963 6709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 12:30:21.154093 6709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "d437cfac39e64c909043a1c57af6668f"
format_stamp: "Formatted at 2026-06-19 12:30:21 on dist-test-slave-fg3h"
server_key: "d5716c29bd498381d019d2586ec08739"
server_key_iv: "dcc607c78a875be32877c99b4ee57708"
server_key_version: "encryptionkey@0"
I20260619 12:30:21.154210 6709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:21.171677 6709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:21.175176 6709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:21.175374 6709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:21.175963 6709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:21.176774 6709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:21.176843 6709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:21.176915 6709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:21.176967 6709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:21.185446 6709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34003
I20260619 12:30:21.185467 6837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34003 every 8 connection(s)
I20260619 12:30:21.186334 6709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:30:21.187443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 6709
I20260619 12:30:21.187551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:30:21.187872 392 external_mini_cluster.cc:1428] Setting key ff5b46039763a9abfa33f87244eaad13
Jun 19 12:30:21 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260619 12:30:21.202134 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.188280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58789 (local address 127.0.98.62:43543)
0619 12:30:21.188444 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:21.188450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:21.189449 (+ 999us) server_negotiation.cc:407] Connection header received
0619 12:30:21.191414 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:21.191417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:21.191467 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:21.191546 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:21.193015 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.193967 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.195680 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.195922 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.196039 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.198491 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:21.198495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:21.198498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:21.198530 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:21.199999 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.200513 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.200519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.200521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.200604 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.200946 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.200951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.200953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.201210 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:21.201317 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:21.201852 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 12:30:21.201966 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:30:21.203059 6840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.188753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.2:58789)
0619 12:30:21.189254 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:21.189312 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:21.191193 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:21.191723 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:21.191730 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:21.192192 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:21.192874 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.192890 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.194149 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.194153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.195501 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.195511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.196076 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.196080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.196258 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.196858 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:21.196874 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:21.198321 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:21.200142 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.200148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.200162 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.200413 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.200711 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.200714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.200716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.200853 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.201314 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:21.201320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:21.201615 (+ 295us) client_negotiation.cc:769] Sending connection context
0619 12:30:21.201810 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 12:30:21.202141 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":101,"threads_started":1}
I20260619 12:30:21.204186 6838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43543
I20260619 12:30:21.204442 6838 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:21.204921 6838 heartbeater.cc:511] Master 127.0.98.62:43543 requested a full tablet report, sending...
I20260619 12:30:21.206044 6513 ts_manager.cc:194] Registered new tserver with Master: d437cfac39e64c909043a1c57af6668f (127.0.98.2:34003)
I20260619 12:30:21.206929 6513 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 12:30:21.216979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43543
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:21.329576 6843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:21.329874 6843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:21.329917 6843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:21.334075 6843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:21.334133 6843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:21.334193 6843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:30:21.339349 6843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:21.340425 6843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:21.341518 6843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:21.351059 6848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:21.351068 6849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:21.351110 6843 server_base.cc:1034] running on GCE node
W20260619 12:30:21.351073 6851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:21.351727 6843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:21.352384 6843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:21.353565 6843 hybrid_clock.cc:648] HybridClock initialized: now 1781872221353534 us; error 44 us; skew 500 ppm
Jun 19 12:30:21 dist-test-slave-fg3h krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:21.356340 6843 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 12:30:21.357291 6843 webserver.cc:460] Webserver started at http://127.0.98.3:40445/ using document root <none> and password file <none>
I20260619 12:30:21.357834 6843 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:21.357892 6843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:21.358111 6843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:21.360159 6843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "ecb3e17ffe404886b679448d181926dc"
format_stamp: "Formatted at 2026-06-19 12:30:21 on dist-test-slave-fg3h"
server_key: "ab3b59e26c969bee91e7b641c3d39870"
server_key_iv: "e6ae32b17b001bba78af99949333310a"
server_key_version: "encryptionkey@0"
I20260619 12:30:21.360718 6843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ecb3e17ffe404886b679448d181926dc"
format_stamp: "Formatted at 2026-06-19 12:30:21 on dist-test-slave-fg3h"
server_key: "ab3b59e26c969bee91e7b641c3d39870"
server_key_iv: "e6ae32b17b001bba78af99949333310a"
server_key_version: "encryptionkey@0"
I20260619 12:30:21.364354 6843 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 12:30:21.366757 6858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:21.368039 6843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:30:21.368134 6843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "ecb3e17ffe404886b679448d181926dc"
format_stamp: "Formatted at 2026-06-19 12:30:21 on dist-test-slave-fg3h"
server_key: "ab3b59e26c969bee91e7b641c3d39870"
server_key_iv: "e6ae32b17b001bba78af99949333310a"
server_key_version: "encryptionkey@0"
I20260619 12:30:21.368260 6843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:21.400707 6843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:21.404114 6843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:21.404310 6843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:21.404899 6843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:21.405755 6843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:21.405808 6843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:21.405875 6843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:21.405915 6843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:21.414912 6843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34403
I20260619 12:30:21.414937 6971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34403 every 8 connection(s)
I20260619 12:30:21.415772 6843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:30:21.423652 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 6843
I20260619 12:30:21.423822 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:30:21.424111 392 external_mini_cluster.cc:1428] Setting key 811173c846bcb1c4bbcd9c6be9f9b25a
Jun 19 12:30:21 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:30:21.429667 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.417512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32973 (local address 127.0.98.62:43543)
0619 12:30:21.417657 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:21.417661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:21.418505 (+ 844us) server_negotiation.cc:407] Connection header received
0619 12:30:21.419809 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:21.419811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:21.419857 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:21.419941 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:21.421581 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.422120 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.423297 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.423531 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.423669 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.426420 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:21.426423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:21.426426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:21.426447 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:21.427931 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.428383 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.428386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.428387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.428456 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.428733 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.428735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.428736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.428891 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:21.428963 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:21.429467 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 12:30:21.429553 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 12:30:21.430550 6974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.417902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.3:32973)
0619 12:30:21.418348 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:21.418401 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:21.419588 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:21.420189 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:21.420202 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:21.420817 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:21.421446 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.421460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.422256 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.422259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.423167 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.423176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.423712 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.423716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.423934 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.424626 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:21.424642 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:21.426278 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:21.428033 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.428037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.428048 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.428301 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.428529 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.428531 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.428533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.428656 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.428975 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:21.428979 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:21.429279 (+ 300us) client_negotiation.cc:769] Sending connection context
0619 12:30:21.429432 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 12:30:21.429669 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":101,"threads_started":1}
I20260619 12:30:21.431640 6972 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43543
I20260619 12:30:21.431905 6972 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:21.432369 6972 heartbeater.cc:511] Master 127.0.98.62:43543 requested a full tablet report, sending...
I20260619 12:30:21.433411 6513 ts_manager.cc:194] Registered new tserver with Master: ecb3e17ffe404886b679448d181926dc (127.0.98.3:34403)
I20260619 12:30:21.433979 6513 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32973
I20260619 12:30:21.437965 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260619 12:30:21.464460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34171
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43543,127.0.98.61:34171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:21.582119 6977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:21.582376 6977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:21.582414 6977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:21.586402 6977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:21.586460 6977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:21.586480 6977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:21.586496 6977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:21.586511 6977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:30:21.591799 6977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43543,127.0.98.61:34171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
W20260619 12:30:21.591979 6977 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:43543,127.0.98.61:34171'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 12:30:21.592957 6977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:21.594290 6977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:30:21.602985 6977 server_base.cc:1034] running on GCE node
W20260619 12:30:21.602901 6985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:21.602905 6982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:21.602905 6983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:21.603703 6977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:21.604825 6977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:21.605995 6977 hybrid_clock.cc:648] HybridClock initialized: now 1781872221605977 us; error 31 us; skew 500 ppm
Jun 19 12:30:21 dist-test-slave-fg3h krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:21.608757 6977 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 12:30:21.609684 6977 webserver.cc:460] Webserver started at http://127.0.98.61:36707/ using document root <none> and password file <none>
I20260619 12:30:21.610266 6977 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:21.610328 6977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:21.610539 6977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:21.612522 6977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/data/instance:
uuid: "d3a7477e1f1e4b52bdf624890ad3c070"
format_stamp: "Formatted at 2026-06-19 12:30:21 on dist-test-slave-fg3h"
server_key: "0656aadbb9ed9f5b2317959486acda77"
server_key_iv: "0d20660bfe358747308df6196a0395cd"
server_key_version: "encryptionkey@0"
I20260619 12:30:21.613225 6977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/wal/instance:
uuid: "d3a7477e1f1e4b52bdf624890ad3c070"
format_stamp: "Formatted at 2026-06-19 12:30:21 on dist-test-slave-fg3h"
server_key: "0656aadbb9ed9f5b2317959486acda77"
server_key_iv: "0d20660bfe358747308df6196a0395cd"
server_key_version: "encryptionkey@0"
I20260619 12:30:21.616977 6977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:30:21.619410 6992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:21.620795 6977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:30:21.620939 6977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/wal
uuid: "d3a7477e1f1e4b52bdf624890ad3c070"
format_stamp: "Formatted at 2026-06-19 12:30:21 on dist-test-slave-fg3h"
server_key: "0656aadbb9ed9f5b2317959486acda77"
server_key_iv: "0d20660bfe358747308df6196a0395cd"
server_key_version: "encryptionkey@0"
I20260619 12:30:21.621049 6977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:21.635418 6977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:21.638293 6977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:21.638484 6977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:21.645152 6977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34171
I20260619 12:30:21.645175 7044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34171 every 8 connection(s)
I20260619 12:30:21.646060 6977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/data/info.pb
I20260619 12:30:21.648692 7045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:21.650972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 6977
I20260619 12:30:21.652904 7045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:30:21 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 12:30:21 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 12:30:21.664575 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.654941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35662 (local address 127.0.98.62:43543)
0619 12:30:21.655073 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:21.655077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:21.655092 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:30:21.655135 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:21.655138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:21.655189 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:21.655283 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:21.656097 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.656630 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.657919 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.658221 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.658342 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.660856 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:21.660861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:21.660866 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:21.660897 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:21.662558 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.663055 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.663059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.663061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.663152 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.663513 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.663517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.663518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.663789 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:21.663918 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:21.664276 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 12:30:21.664391 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 12:30:21.665412 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.654558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:35662)
0619 12:30:21.654892 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:21.654937 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:21.655038 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:21.655336 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:21.655341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:21.655771 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:21.655961 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.655974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.656755 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.656758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.657786 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.657795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.658362 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.658365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.658436 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.659042 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:21.659057 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:21.660658 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:21.662699 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.662708 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.662716 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.662946 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.663253 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.663255 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.663257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.663406 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.663906 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:21.663912 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:21.664061 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 12:30:21.664232 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 12:30:21.664526 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":59,"spinlock_wait_cycles":4864,"thread_start_us":124,"threads_started":1}
I20260619 12:30:21.665416 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.652530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46338 (local address 127.0.98.61:34171)
0619 12:30:21.652948 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:21.652962 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:21.652995 (+ 33us) server_negotiation.cc:407] Connection header received
0619 12:30:21.653754 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:21.653774 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:21.654083 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:21.654401 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:21.655286 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.656505 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.657406 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.658025 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.658133 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.661321 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:21.661326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:21.661334 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:21.661355 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:21.662790 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.663241 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.663248 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.663255 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.663370 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.663706 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.663710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.663712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.664052 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:21.664242 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:21.664467 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 12:30:21.664605 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":291,"spinlock_wait_cycles":2816,"thread_start_us":137,"threads_started":1}
I20260619 12:30:21.666824 7045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:30:21 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 12:30:21.675233 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.667305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34171 (local address 127.0.0.1:46340)
0619 12:30:21.667415 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:21.667427 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:21.667507 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:21.667780 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:21.667783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:21.667932 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:21.668006 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.668010 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.668701 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.668707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.669750 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.669758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.670192 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.670195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.670286 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.670812 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:21.670823 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:21.672017 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:21.673900 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.673903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.673904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.674081 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.674380 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.674383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.674384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.674465 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.674867 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:21.674870 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:21.674940 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 12:30:21.675026 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 12:30:21.675114 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 12:30:21.675355 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.667445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46340 (local address 127.0.98.61:34171)
0619 12:30:21.667561 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:21.667565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:21.667577 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:30:21.667605 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:21.667607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:21.667686 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:21.667769 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:21.668107 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.668590 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.669862 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.670088 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.670197 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.672143 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:21.672147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:21.672149 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:21.672170 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:21.673792 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.674192 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.674195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.674197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.674273 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.674557 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.674560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.674561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.674758 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:21.674846 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:21.675086 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 12:30:21.675192 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 12:30:21.678001 7045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: Bootstrap starting.
I20260619 12:30:21.680087 7045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:21.680825 7045 log.cc:824] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:21.682669 7045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: No bootstrap required, opened a new log
I20260619 12:30:21.685636 7045 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } }
I20260619 12:30:21.685818 7045 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:21.685894 7045 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a7477e1f1e4b52bdf624890ad3c070, State: Initialized, Role: FOLLOWER
I20260619 12:30:21.686296 7045 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } }
I20260619 12:30:21.686802 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } } }
I20260619 12:30:21.686937 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:30:21.687542 7045 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 12:30:21.692288 7062 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:30:21.692374 7062 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:30:21.693790 6977 master_runner.cc:428] Detected that this master d3a7477e1f1e4b52bdf624890ad3c070 is joining an existing cluster
I20260619 12:30:21.693840 6977 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:34171
I20260619 12:30:21.704654 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.696388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46342 (local address 127.0.98.61:34171)
0619 12:30:21.696487 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:21.696491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:21.697006 (+ 515us) server_negotiation.cc:407] Connection header received
0619 12:30:21.697155 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:21.697160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:21.697204 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:21.697296 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:21.697803 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.698523 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.699448 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.699701 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.699815 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.701046 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:21.701050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:21.701055 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:21.701106 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:21.702778 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.703246 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.703251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.703253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.703332 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.703593 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.703598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.703600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.704034 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:21.704117 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:21.704354 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 12:30:21.704489 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":41}
I20260619 12:30:21.704669 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.696326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34171 (local address 127.0.0.1:46342)
0619 12:30:21.696938 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:21.696950 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:21.697051 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:21.697334 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:21.697337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:21.697544 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:21.697673 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.697682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.698643 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.698645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.699296 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.699303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.699827 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.699831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.699948 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.700465 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:21.700480 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:21.700925 (+ 445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:21.702892 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.702898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.702903 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.703122 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.703428 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.703431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.703433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.703507 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.704106 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:21.704109 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:21.704199 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 12:30:21.704302 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 12:30:21.704439 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":60,"thread_start_us":77,"threads_started":1}
I20260619 12:30:21.705351 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.695952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35672 (local address 127.0.98.62:43543)
0619 12:30:21.696077 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:21.696082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:21.696699 (+ 617us) server_negotiation.cc:407] Connection header received
0619 12:30:21.696826 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:21.696828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:21.696869 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:21.696964 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:21.697575 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.698455 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.699736 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.700034 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.700166 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.701710 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:21.701714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:21.701717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:21.701752 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:21.703618 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.704027 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.704030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.704032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.704118 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.704427 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.704430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.704432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.704627 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:21.704693 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:21.705056 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 12:30:21.705199 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:30:21.705881 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.695894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:35672)
0619 12:30:21.696601 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:21.696616 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:21.696715 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:21.696979 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:21.696982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:21.697296 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:21.697465 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.697471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.698571 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.698575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.699592 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.699602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.700440 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.700445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.700518 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.700989 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:21.701009 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:21.701530 (+ 521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:21.703716 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.703719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.703721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.703930 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.704228 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.704231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.704234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.704322 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.704777 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:21.704780 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:21.704880 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 12:30:21.705544 (+ 664us) client_negotiation.cc:240] Negotiation successful
0619 12:30:21.705704 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":37,"thread_start_us":106,"threads_started":1}
Jun 19 12:30:21 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 12:30:21.719310 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.709949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34171 (local address 127.0.0.1:46358)
0619 12:30:21.710319 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:21.710372 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:21.710502 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:21.710791 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:21.710796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:21.711222 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:21.711346 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.711353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.712008 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.712011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.713069 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.713103 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.713526 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.713529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.713650 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.714140 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:21.714155 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:21.715668 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:21.717442 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.717451 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.717457 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.717771 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.718078 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.718081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.718083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.718190 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.718646 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:21.718653 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:21.718786 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 12:30:21.718939 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 12:30:21.719134 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":109,"threads_started":1}
I20260619 12:30:21.719372 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.709991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46358 (local address 127.0.98.61:34171)
0619 12:30:21.710116 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:21.710120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:21.710457 (+ 337us) server_negotiation.cc:407] Connection header received
0619 12:30:21.710624 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:21.710626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:21.710667 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:21.710732 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:21.711452 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.711913 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.713206 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.713423 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.713502 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.715810 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:21.715814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:21.715818 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:21.715844 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:21.717321 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.717858 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.717863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.717865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.717968 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.718308 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.718312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.718313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.718510 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:21.718595 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:21.719018 (+ 423us) server_negotiation.cc:299] Negotiation successful
0619 12:30:21.719135 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:30:21.721145 6513 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:34171: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 34171
}
attrs {
promote: true
}
}
dest_uuid: "5163b08470e840a98fb193929dc7ad25"
cas_config_opid_index: -1
I20260619 12:30:21.721778 6513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: true } }
I20260619 12:30:21.723412 7073 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d3a7477e1f1e4b52bdf624890ad3c070 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: true } } }
I20260619 12:30:21.724011 7019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:21.725399 7073 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:34171
I20260619 12:30:21.725656 7019 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 1 FOLLOWER]: Refusing update from remote peer 5163b08470e840a98fb193929dc7ad25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 12:30:21.725890 6977 master.cc:537] Master@127.0.98.61:34171 shutting down...
I20260619 12:30:21.725848 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5163b08470e840a98fb193929dc7ad25" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: true } } }
I20260619 12:30:21.725939 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:21.726570 7072 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 12:30:21.728451 6499 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 12:30:21.729504 6499 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 -> Peer d3a7477e1f1e4b52bdf624890ad3c070 (127.0.98.61:34171): Couldn't send request to peer d3a7477e1f1e4b52bdf624890ad3c070. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 12:30:21.730644 6977 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:21.730940 6977 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:21.731067 6977 tablet_replica.cc:330] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: stopping tablet replica
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260619 12:30:21.746402 6977 master.cc:552] Master@127.0.98.61:34171 shutdown complete.
I20260619 12:30:21.746452 6977 master_runner.cc:305] Clearing existing system tablet
I20260619 12:30:21.746726 6977 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:21.746790 6977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:21.748487 6977 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:30:21.749476 7079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:21.749683 6977 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:21.749823 6977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:30:21.749858 6977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/wal
uuid: "d3a7477e1f1e4b52bdf624890ad3c070"
format_stamp: "Formatted at 2026-06-19 12:30:21 on dist-test-slave-fg3h"
server_key: "0656aadbb9ed9f5b2317959486acda77"
server_key_iv: "0d20660bfe358747308df6196a0395cd"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260619 12:30:21.752187 6977 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:21.752516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32971
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43543,127.0.98.61:34171,127.0.98.60:32971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:30:21.754904 6977 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 12:30:21.754990 6977 log.cc:1196] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 12:30:21.755295 6977 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: Deleting consensus metadata
I20260619 12:30:21.755458 6977 master_runner.cc:315] Copying system tablet from 127.0.98.62:43543
I20260619 12:30:21.756562 6977 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43543
I20260619 12:30:21.766294 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.757170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:35686)
0619 12:30:21.757439 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:21.757454 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:21.757571 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:21.758008 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:21.758011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:21.758309 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:21.758433 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.758440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.759644 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.759647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.760553 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.760562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.761293 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.761296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.761393 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.762016 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:21.762030 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:21.762430 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:21.764606 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.764609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.764611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.764870 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.765193 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.765196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.765198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.765302 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.765735 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:21.765738 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:21.765834 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 12:30:21.765940 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 12:30:21.766075 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":95,"threads_started":1}
I20260619 12:30:21.766350 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.757575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35686 (local address 127.0.98.62:43543)
0619 12:30:21.757734 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:21.757741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:21.757762 (+ 21us) server_negotiation.cc:407] Connection header received
0619 12:30:21.757817 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:21.757823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:21.757885 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:21.757990 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:21.758567 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.759510 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.760691 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.761170 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.761301 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.762585 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:21.762592 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:21.762598 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:21.762634 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:21.764489 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.764967 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.764970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.764972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.765055 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.765395 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.765398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.765400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.765625 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:21.765730 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:21.766082 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 12:30:21.766197 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 12:30:21.767220 6533 tablet_copy_service.cc:140] P 5163b08470e840a98fb193929dc7ad25: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d3a7477e1f1e4b52bdf624890ad3c070 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:35686)
I20260619 12:30:21.767402 6533 tablet_copy_service.cc:161] P 5163b08470e840a98fb193929dc7ad25: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d3a7477e1f1e4b52bdf624890ad3c070 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:35686: session id = d3a7477e1f1e4b52bdf624890ad3c070-00000000000000000000000000000000
I20260619 12:30:21.769188 6533 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 12:30:21.770854 6977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 12:30:21.773151 6977 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: tablet copy: Starting download of 0 data blocks...
I20260619 12:30:21.773347 6977 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: tablet copy: Starting download of 1 WAL segments...
I20260619 12:30:21.775055 6977 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 12:30:21.776795 6533 tablet_copy_service.cc:342] P 5163b08470e840a98fb193929dc7ad25: Request end of tablet copy session d3a7477e1f1e4b52bdf624890ad3c070-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:35686
I20260619 12:30:21.776913 6533 tablet_copy_service.cc:433] P 5163b08470e840a98fb193929dc7ad25: ending tablet copy session d3a7477e1f1e4b52bdf624890ad3c070-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d3a7477e1f1e4b52bdf624890ad3c070
I20260619 12:30:21.783913 6977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:21.785614 7087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:21.786121 7090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:21.789533 7088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:21.789911 6977 server_base.cc:1034] running on GCE node
I20260619 12:30:21.790241 6977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:21.790540 6977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:21.791690 6977 hybrid_clock.cc:648] HybridClock initialized: now 1781872221791678 us; error 31 us; skew 500 ppm
Jun 19 12:30:21 dist-test-slave-fg3h krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:21.793314 6977 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 12:30:21.793859 6977 webserver.cc:460] Webserver started at http://127.0.98.61:37975/ using document root <none> and password file <none>
I20260619 12:30:21.794054 6977 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:21.794097 6977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:21.795398 6977 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:30:21.796556 7097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:21.796793 6977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:30:21.796866 6977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/wal
uuid: "d3a7477e1f1e4b52bdf624890ad3c070"
format_stamp: "Formatted at 2026-06-19 12:30:21 on dist-test-slave-fg3h"
server_key: "0656aadbb9ed9f5b2317959486acda77"
server_key_iv: "0d20660bfe358747308df6196a0395cd"
server_key_version: "encryptionkey@0"
I20260619 12:30:21.796943 6977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:21.808084 6977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:21.810150 6977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:21.815093 6977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34171
I20260619 12:30:21.815661 6977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-1/data/info.pb
I20260619 12:30:21.817349 7149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34171 every 8 connection(s)
I20260619 12:30:21.817525 7150 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:30:21.818502 7150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: Bootstrap starting.
I20260619 12:30:21.828455 7150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:30:21.828785 7150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: Bootstrap complete.
I20260619 12:30:21.829381 7150 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: true } }
I20260619 12:30:21.829519 7150 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d3a7477e1f1e4b52bdf624890ad3c070, State: Initialized, Role: LEARNER
I20260619 12:30:21.829614 7150 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: true } }
I20260619 12:30:21.829983 7153 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: true } } }
I20260619 12:30:21.830102 7150 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:21.830183 7153 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [sys.catalog]: This master's current role is: LEARNER
I20260619 12:30:21.833693 7164 catalog_manager.cc:1259] Loaded cluster ID: 1e3227015b29436ab0201074af2dcaf3
I20260619 12:30:21.833757 7164 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: loading cluster ID for follower catalog manager: success
I20260619 12:30:21.835873 7164 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: acquiring CA information for follower catalog manager: success
I20260619 12:30:21.837388 7164 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 12:30:21.883940 7080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:21.884239 7080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:21.884311 7080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:21.888441 7080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:21.888517 7080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:21.888554 7080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:21.888610 7080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:21.888644 7080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:30:21.894124 7080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43543,127.0.98.61:34171,127.0.98.60:32971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:21.895226 7080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:21.896355 7080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:21.905272 7170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:21.905367 7080 server_base.cc:1034] running on GCE node
W20260619 12:30:21.905262 7172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:21.905264 7169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:21.906142 7080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:21.907239 7080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:21.908407 7080 hybrid_clock.cc:648] HybridClock initialized: now 1781872221908372 us; error 48 us; skew 500 ppm
Jun 19 12:30:21 dist-test-slave-fg3h krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:21.911316 7080 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 12:30:21.912223 7080 webserver.cc:460] Webserver started at http://127.0.98.60:41345/ using document root <none> and password file <none>
I20260619 12:30:21.912783 7080 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:21.912873 7080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:21.913159 7080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:21.915355 7080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/data/instance:
uuid: "6520e328cbfb4ba8aefb2874c02d3dec"
format_stamp: "Formatted at 2026-06-19 12:30:21 on dist-test-slave-fg3h"
server_key: "51ff74d230142091dc68b51823548161"
server_key_iv: "f768b23f21043eb6b4c791352c3c32e7"
server_key_version: "encryptionkey@0"
I20260619 12:30:21.915930 7080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/wal/instance:
uuid: "6520e328cbfb4ba8aefb2874c02d3dec"
format_stamp: "Formatted at 2026-06-19 12:30:21 on dist-test-slave-fg3h"
server_key: "51ff74d230142091dc68b51823548161"
server_key_iv: "f768b23f21043eb6b4c791352c3c32e7"
server_key_version: "encryptionkey@0"
I20260619 12:30:21.919648 7080 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 12:30:21.922160 7179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:21.923381 7080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:21.923527 7080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/wal
uuid: "6520e328cbfb4ba8aefb2874c02d3dec"
format_stamp: "Formatted at 2026-06-19 12:30:21 on dist-test-slave-fg3h"
server_key: "51ff74d230142091dc68b51823548161"
server_key_iv: "f768b23f21043eb6b4c791352c3c32e7"
server_key_version: "encryptionkey@0"
I20260619 12:30:21.923642 7080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:21.942219 7080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:21.945727 7080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:21.945932 7080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:21.953198 7080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:32971
I20260619 12:30:21.953491 7231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:32971 every 8 connection(s)
I20260619 12:30:21.954216 7080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/data/info.pb
I20260619 12:30:21.957032 7232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:21.959234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 7080
I20260619 12:30:21.962189 7232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:30:21 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 12:30:21 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:30:21.974867 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.964030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35690 (local address 127.0.98.62:43543)
0619 12:30:21.964144 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:21.964148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:21.964508 (+ 360us) server_negotiation.cc:407] Connection header received
0619 12:30:21.964723 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:21.964727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:21.964786 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:21.964876 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:21.966257 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.967081 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.968311 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.968530 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.968642 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.971135 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:21.971138 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:21.971140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:21.971164 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:21.973118 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.973564 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.973568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.973569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.973655 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.973926 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.973931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.973933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.974110 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:21.974216 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:21.974565 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 12:30:21.974680 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:30:21.975699 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.960644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.0.98.60:32971)
0619 12:30:21.961041 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:21.961056 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:21.961147 (+ 91us) server_negotiation.cc:407] Connection header received
0619 12:30:21.961841 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:21.961860 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:21.962190 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:21.962524 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:21.963889 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.965419 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.966620 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.967111 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.967348 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.970354 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:21.970360 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:21.970370 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:21.970392 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:21.972505 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.973138 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.973145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.973151 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.973275 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.973580 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.973584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.973586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.974027 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:21.974223 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:21.974493 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 12:30:21.974830 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":104,"threads_started":1}
I20260619 12:30:21.975700 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.963898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:35690)
0619 12:30:21.964384 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:21.964432 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:21.964604 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:21.964921 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:21.964927 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:21.965915 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:21.966067 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.966075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.967245 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.967251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.968174 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.968184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.968693 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.968698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.968805 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.969430 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:21.969448 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:21.970984 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:21.973211 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.973219 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.973233 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.973451 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.973751 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.973754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.973758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.973835 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.974213 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:21.974219 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:21.974354 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 12:30:21.974514 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 12:30:21.974939 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":124,"thread_start_us":92,"threads_started":1}
I20260619 12:30:21.977656 7232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:30:21 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 12:30:21.985980 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.978143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34171 (local address 127.0.0.1:46368)
0619 12:30:21.978472 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:21.978484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:21.978545 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:21.978780 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:21.978782 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:21.978934 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:21.979031 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.979035 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.979945 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.979948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.980496 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.980504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.980909 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.980912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.980978 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.981430 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:21.981442 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:21.982706 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:21.984580 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.984583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.984584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.984809 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.985052 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.985054 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.985055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.985216 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.985548 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:21.985550 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:21.985674 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 12:30:21.985758 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 12:30:21.985860 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269}
I20260619 12:30:21.986029 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.978269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.0.98.61:34171)
0619 12:30:21.978541 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:21.978545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:21.978558 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:30:21.978613 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:21.978615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:21.978654 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:21.978721 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:21.979143 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.979843 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.980594 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.980820 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.980894 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.982899 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:21.982902 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:21.982904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:21.982926 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:21.984470 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.984897 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.984899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.984901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.984973 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.985296 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.985298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.985299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.985469 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:21.985545 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:21.985814 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 12:30:21.985911 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":84,"threads_started":1}
I20260619 12:30:21.986841 6704 heartbeater.cc:503] Master 127.0.98.62:43543 was elected leader, sending a full tablet report...
I20260619 12:30:21.986954 7232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:30:21 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872221, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 12:30:21.995627 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.987538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37970 (local address 127.0.98.60:32971)
0619 12:30:21.987637 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:21.987640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:21.987652 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:30:21.987736 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:21.987738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:21.987771 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:21.987839 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:21.988219 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.989448 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.990231 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.990450 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.990556 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.992416 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:21.992419 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:21.992421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:21.992441 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:21.993936 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.994367 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.994370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.994371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.994517 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:21.994824 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:21.994826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:21.994827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:21.994989 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:21.995095 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:21.995322 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 12:30:21.995472 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":38}
I20260619 12:30:21.995627 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:21.987454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32971 (local address 127.0.0.1:37970)
0619 12:30:21.987571 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:21.987585 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:21.987655 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:21.987858 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:21.987861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:21.988026 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:21.988111 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.988116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.989567 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.989570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.990081 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:21.990089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:21.990551 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:21.990554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:21.990622 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:21.991036 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:21.991048 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:21.992285 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:21.994038 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.994041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.994042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.994280 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.994607 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:21.994610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:21.994612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:21.994691 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:21.995084 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:21.995087 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:21.995172 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 12:30:21.995284 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 12:30:21.995418 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 12:30:21.998993 7232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: Bootstrap starting.
I20260619 12:30:22.001296 7232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:22.001971 7232 log.cc:824] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:22.003923 7232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: No bootstrap required, opened a new log
I20260619 12:30:22.006951 7232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } }
I20260619 12:30:22.007170 7232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:22.007205 7232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6520e328cbfb4ba8aefb2874c02d3dec, State: Initialized, Role: FOLLOWER
I20260619 12:30:22.007637 7232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } }
I20260619 12:30:22.008113 7238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } } }
I20260619 12:30:22.008285 7238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:30:22.008845 7232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [sys.catalog]: configured and running, proceeding with master startup.
W20260619 12:30:22.014125 7249 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:30:22.014241 7249 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:30:22.017839 7080 master_runner.cc:428] Detected that this master 6520e328cbfb4ba8aefb2874c02d3dec is joining an existing cluster
I20260619 12:30:22.017908 7080 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:32971
I20260619 12:30:22.029222 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 6575
I20260619 12:30:22.036550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42353
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43543
--webserver_port=37057
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43543,127.0.98.61:34171
--tserver_master_addrs=127.0.98.62:43543,127.0.98.61:34171,127.0.98.60:32971 with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:30:22.058334 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.020804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34171 (local address 127.0.0.1:46380)
0619 12:30:22.021660 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.021671 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.021840 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.022602 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.022605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.022807 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:22.022905 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.022911 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.023910 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.023913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.024907 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.024914 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.025607 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.025613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.025728 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.053767 (+ 28039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:22.053798 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:22.054238 (+ 440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:22.056515 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.056519 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.056521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.056765 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.057199 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.057202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.057204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.057288 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.057747 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:22.057750 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:22.057858 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.057959 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.058130 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":111,"thread_start_us":95,"threads_started":1}
I20260619 12:30:22.058527 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.021073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.0.98.62:43543)
0619 12:30:22.021234 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.021239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.021272 (+ 33us) server_negotiation.cc:407] Connection header received
0619 12:30:22.021316 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.021319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.021368 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.021446 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.022576 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.023450 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.025198 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.025551 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.025649 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.054206 (+ 28557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.054213 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.054218 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.054256 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.056524 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.057042 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.057045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.057047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.057164 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.057495 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.057498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.057500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.057711 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.057796 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.058191 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.058344 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 12:30:22.058984 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.020883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46380 (local address 127.0.98.61:34171)
0619 12:30:22.020997 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.021002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.021730 (+ 728us) server_negotiation.cc:407] Connection header received
0619 12:30:22.021960 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.021964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.022015 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.022067 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.023030 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.023783 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.025031 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.025387 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.025459 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.054416 (+ 28957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.054421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.054424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.054459 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.056363 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.056950 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.056953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.056955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.057044 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.057384 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.057387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.057389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.057619 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.057706 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.058687 (+ 981us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.058827 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 12:30:22.058997 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.020379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:35696)
0619 12:30:22.021054 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.021069 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.021195 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.021571 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.021575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.022301 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:22.022427 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.022435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.023590 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.023593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.025027 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.025036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.025821 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.025827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.025951 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.026418 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:22.026433 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:22.053947 (+ 27514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:22.056680 (+ 2733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.056683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.056685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.056931 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.057304 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.057307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.057309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.057400 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.057835 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:22.057838 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:22.057949 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.058032 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.058816 (+ 784us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":683,"spinlock_wait_cycles":2304,"thread_start_us":104,"threads_started":1}
I20260619 12:30:22.064668 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.020475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32971 (local address 127.0.0.1:37980)
0619 12:30:22.021333 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.021343 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.021448 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.021779 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.021781 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.022012 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:22.022599 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.022605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.023541 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.023543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.024425 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.024434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.025367 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.025370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.025456 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.026068 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:22.026081 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:22.055628 (+ 29547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:22.063145 (+ 7517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.063148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.063150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.063397 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.063658 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.063661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.063663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.063731 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.064123 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:22.064125 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:22.064196 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.064289 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.064429 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260619 12:30:22.064675 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.020486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37980 (local address 127.0.98.60:32971)
0619 12:30:22.020599 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.020604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.021415 (+ 811us) server_negotiation.cc:407] Connection header received
0619 12:30:22.021541 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.021544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.021583 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.021651 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.022706 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.023433 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.024534 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.025150 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.025233 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.060348 (+ 35115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.060353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.060356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.060380 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.063022 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.063494 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.063497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.063499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.063569 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.063810 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.063813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.063814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.064005 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.064069 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.064376 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.064496 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":47}
Jun 19 12:30:22 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 12:30:22.074271 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.063550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32971 (local address 127.0.0.1:37990)
0619 12:30:22.063665 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.063680 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.063772 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.065262 (+ 1490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.065265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.065464 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:22.065573 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.065579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.066760 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.066764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.067594 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.067605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.068205 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.068209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.068321 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.068854 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:22.068868 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:22.070419 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:22.072607 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.072610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.072613 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.072853 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.073220 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.073223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.073225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.073315 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.073760 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:22.073763 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:22.073863 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.073942 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.074088 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 12:30:22.074971 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.063780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37990 (local address 127.0.98.60:32971)
0619 12:30:22.064943 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.064951 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.064964 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:30:22.065027 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.065030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.065078 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.065706 (+ 628us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.065804 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.066630 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.067724 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.068078 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.068183 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.070559 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.070563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.070566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.070590 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.072485 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.072979 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.072982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.072984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.073073 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.073440 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.073442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.073444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.073653 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.074588 (+ 935us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.074675 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.074795 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1091,"thread_start_us":139,"threads_started":1}
I20260619 12:30:22.075894 6513 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:32971: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 32971
}
attrs {
promote: true
}
}
dest_uuid: "5163b08470e840a98fb193929dc7ad25"
cas_config_opid_index: 5
I20260619 12:30:22.076484 6513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: true } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: true } }
I20260619 12:30:22.077628 7072 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6520e328cbfb4ba8aefb2874c02d3dec (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: true } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: true } } }
I20260619 12:30:22.078073 7206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:22.079164 7072 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:32971
I20260619 12:30:22.079376 7074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5163b08470e840a98fb193929dc7ad25" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: true } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: true } } }
I20260619 12:30:22.079447 7206 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 1 FOLLOWER]: Refusing update from remote peer 5163b08470e840a98fb193929dc7ad25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 12:30:22.079492 7074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:22.080171 7080 master.cc:537] Master@127.0.98.60:32971 shutting down...
I20260619 12:30:22.080142 7074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 12:30:22.082012 6500 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 -> Peer 6520e328cbfb4ba8aefb2874c02d3dec (127.0.98.60:32971): Couldn't send request to peer 6520e328cbfb4ba8aefb2874c02d3dec. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 12:30:22.084203 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.077617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34171 (local address 127.0.0.1:46388)
0619 12:30:22.077848 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.077862 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.077936 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.079839 (+ 1903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.079842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.079863 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:22.079958 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.079964 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.081118 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.081122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.082532 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.082543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.083771 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.083775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.083874 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.083920 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.083994 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.084072 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260619 12:30:22.084659 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.079438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.0.98.61:34171)
0619 12:30:22.079563 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.079568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.079582 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:30:22.079632 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.079634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.079692 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.079800 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:22.080094 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.080868 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.082748 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.083454 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.083558 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.083710 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.084467 (+ 757us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.084548 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:30:22.085548 7080 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:22.085561 7124 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 1 LEARNER]: Refusing update from remote peer 5163b08470e840a98fb193929dc7ad25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 12:30:22.085855 7080 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:22.085973 7080 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: stopping tablet replica
I20260619 12:30:22.085987 7074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 12:30:22.087936 7124 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6520e328cbfb4ba8aefb2874c02d3dec (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: true } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: true } } }
I20260619 12:30:22.090267 7153 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5163b08470e840a98fb193929dc7ad25. Latest consensus state: current_term: 1 leader_uuid: "5163b08470e840a98fb193929dc7ad25" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: true } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: true } } }
I20260619 12:30:22.090394 7153 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [sys.catalog]: This master's current role is: LEARNER
I20260619 12:30:22.090482 7153 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5163b08470e840a98fb193929dc7ad25" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: true } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: true } } }
I20260619 12:30:22.090556 7153 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [sys.catalog]: This master's current role is: LEARNER
I20260619 12:30:22.093428 7074 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25: attempting to promote NON_VOTER d3a7477e1f1e4b52bdf624890ad3c070 to VOTER
I20260619 12:30:22.093941 7074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: true } }
I20260619 12:30:22.094827 7080 master.cc:552] Master@127.0.98.60:32971 shutdown complete.
I20260619 12:30:22.094913 7080 master_runner.cc:305] Clearing existing system tablet
I20260619 12:30:22.095191 7080 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:22.095273 7080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:22.095798 7124 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 1 LEARNER]: Refusing update from remote peer 5163b08470e840a98fb193929dc7ad25: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 12:30:22.096292 7073 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 12:30:22.097040 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.095324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32971 (local address 127.0.0.1:37998)
0619 12:30:22.096875 (+ 1551us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.096950 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1470}
I20260619 12:30:22.097940 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.097533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32971 (local address 127.0.0.1:38000)
0619 12:30:22.097799 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.097854 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":94,"threads_started":1}
W20260619 12:30:22.098109 6500 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 -> Peer 6520e328cbfb4ba8aefb2874c02d3dec (127.0.98.60:32971): Couldn't send request to peer 6520e328cbfb4ba8aefb2874c02d3dec. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 12:30:22.098603 7073 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d3a7477e1f1e4b52bdf624890ad3c070 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: true } } }
I20260619 12:30:22.100099 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5163b08470e840a98fb193929dc7ad25" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: true } } }
I20260619 12:30:22.100216 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:22.100346 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5163b08470e840a98fb193929dc7ad25" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: true } } }
I20260619 12:30:22.100422 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:22.101111 7123 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d3a7477e1f1e4b52bdf624890ad3c070 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: true } } }
I20260619 12:30:22.102615 7261 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5163b08470e840a98fb193929dc7ad25" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: true } } }
I20260619 12:30:22.102785 7261 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:30:22.103432 7080 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.000s
I20260619 12:30:22.104403 7264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:22.104686 7080 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:22.104808 7080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:30:22.104843 7080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/wal
uuid: "6520e328cbfb4ba8aefb2874c02d3dec"
format_stamp: "Formatted at 2026-06-19 12:30:21 on dist-test-slave-fg3h"
server_key: "51ff74d230142091dc68b51823548161"
server_key_iv: "f768b23f21043eb6b4c791352c3c32e7"
server_key_version: "encryptionkey@0"
I20260619 12:30:22.106943 7080 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:22.109344 7080 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 12:30:22.109468 7080 log.cc:1196] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 12:30:22.109853 7080 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: Deleting consensus metadata
I20260619 12:30:22.110014 7080 master_runner.cc:315] Copying system tablet from 127.0.98.62:43543
I20260619 12:30:22.111188 7080 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43543
I20260619 12:30:22.126009 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.111723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:35700)
0619 12:30:22.117299 (+ 5576us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.117317 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.117471 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.117827 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.117830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.118080 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:22.118473 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.118483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.119604 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.119608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.120508 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.120518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.121113 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.121118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.121218 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.121822 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:22.121837 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:22.122275 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:22.124371 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.124375 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.124377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.124626 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.124915 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.124918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.124920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.125006 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.125449 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:22.125452 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:22.125551 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.125669 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.125813 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5482,"thread_start_us":93,"threads_started":1}
I20260619 12:30:22.126645 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.111845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35700 (local address 127.0.98.62:43543)
0619 12:30:22.111975 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.111979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.117452 (+ 5473us) server_negotiation.cc:407] Connection header received
0619 12:30:22.117605 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.117608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.117681 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.117783 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.118613 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.119472 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.120636 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.120968 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.121079 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.122522 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.122526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.122529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.122559 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.124247 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.124725 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.124728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.124730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.124823 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.125128 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.125131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.125133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.125353 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.125455 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.125908 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.126041 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:30:22.127542 6533 tablet_copy_service.cc:140] P 5163b08470e840a98fb193929dc7ad25: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6520e328cbfb4ba8aefb2874c02d3dec ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:35700)
I20260619 12:30:22.127648 6533 tablet_copy_service.cc:161] P 5163b08470e840a98fb193929dc7ad25: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6520e328cbfb4ba8aefb2874c02d3dec at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:35700: session id = 6520e328cbfb4ba8aefb2874c02d3dec-00000000000000000000000000000000
I20260619 12:30:22.128767 6533 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 12:30:22.130148 7080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 12:30:22.132532 7080 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: tablet copy: Starting download of 0 data blocks...
I20260619 12:30:22.132716 7080 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: tablet copy: Starting download of 1 WAL segments...
I20260619 12:30:22.134202 7080 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 12:30:22.136001 6533 tablet_copy_service.cc:342] P 5163b08470e840a98fb193929dc7ad25: Request end of tablet copy session 6520e328cbfb4ba8aefb2874c02d3dec-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:35700
I20260619 12:30:22.136088 6533 tablet_copy_service.cc:433] P 5163b08470e840a98fb193929dc7ad25: ending tablet copy session 6520e328cbfb4ba8aefb2874c02d3dec-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6520e328cbfb4ba8aefb2874c02d3dec
I20260619 12:30:22.144297 7080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:22.146363 7274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:22.146379 7271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:22.147046 7080 server_base.cc:1034] running on GCE node
W20260619 12:30:22.150127 7272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:22.150466 7080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:22.150797 7080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:22.151947 7080 hybrid_clock.cc:648] HybridClock initialized: now 1781872222151925 us; error 40 us; skew 500 ppm
Jun 19 12:30:22 dist-test-slave-fg3h krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872222, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:22.153782 7080 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 12:30:22.154253 7080 webserver.cc:460] Webserver started at http://127.0.98.60:38863/ using document root <none> and password file <none>
I20260619 12:30:22.154462 7080 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:22.154503 7080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:22.155938 7080 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 12:30:22.157032 7281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:22.157400 7080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 12:30:22.157477 7080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/wal
uuid: "6520e328cbfb4ba8aefb2874c02d3dec"
format_stamp: "Formatted at 2026-06-19 12:30:21 on dist-test-slave-fg3h"
server_key: "51ff74d230142091dc68b51823548161"
server_key_iv: "f768b23f21043eb6b4c791352c3c32e7"
server_key_version: "encryptionkey@0"
I20260619 12:30:22.157562 7080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:22.167578 7080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:22.169942 7080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:22.175026 7080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:32971
I20260619 12:30:22.175068 7333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:32971 every 8 connection(s)
I20260619 12:30:22.175633 7080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/master-2/data/info.pb
I20260619 12:30:22.177469 7334 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:30:22.178565 7334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: Bootstrap starting.
I20260619 12:30:22.190119 7334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:30:22.190441 7334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: Bootstrap complete.
I20260619 12:30:22.191061 7334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: true } }
I20260619 12:30:22.191208 7334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6520e328cbfb4ba8aefb2874c02d3dec, State: Initialized, Role: LEARNER
I20260619 12:30:22.191310 7334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: true } }
I20260619 12:30:22.191610 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: true } } }
I20260619 12:30:22.191740 7334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:22.191833 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [sys.catalog]: This master's current role is: LEARNER
I20260619 12:30:22.195242 7351 catalog_manager.cc:1259] Loaded cluster ID: 1e3227015b29436ab0201074af2dcaf3
I20260619 12:30:22.195300 7351 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: loading cluster ID for follower catalog manager: success
I20260619 12:30:22.197422 7351 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: acquiring CA information for follower catalog manager: success
W20260619 12:30:22.198112 7258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:22.198457 7258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:22.198526 7258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 12:30:22.198848 7351 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 12:30:22.202936 7258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:22.202993 7258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:22.203094 7258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:30:22.208505 7258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37057
--tserver_master_addrs=127.0.98.62:43543,127.0.98.61:34171,127.0.98.60:32971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:22.208909 6838 heartbeater.cc:503] Master 127.0.98.62:43543 was elected leader, sending a full tablet report...
I20260619 12:30:22.209837 7258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:22.211068 7258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:22.221118 7354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:22.221153 7353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:22.221105 7356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:22.221328 7258 server_base.cc:1034] running on GCE node
I20260619 12:30:22.221817 7258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:22.222483 7258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:22.223676 7258 hybrid_clock.cc:648] HybridClock initialized: now 1781872222223660 us; error 24 us; skew 500 ppm
Jun 19 12:30:22 dist-test-slave-fg3h krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872222, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:22.226718 7258 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 12:30:22.227599 7258 webserver.cc:460] Webserver started at http://127.0.98.1:37057/ using document root <none> and password file <none>
I20260619 12:30:22.228145 7258 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:22.228216 7258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:22.232504 7258 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260619 12:30:22.234699 7363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:22.235960 7258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:30:22.236052 7258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "79139bcb3a2c449996505234878d5ed2"
format_stamp: "Formatted at 2026-06-19 12:30:20 on dist-test-slave-fg3h"
server_key: "932e32ed5860f203555e0004fa4035c1"
server_key_iv: "1cc52a7873fadfb353b9824dff81cd4d"
server_key_version: "encryptionkey@0"
I20260619 12:30:22.236609 7258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:22.296142 7258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:22.299630 7258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:22.299851 7258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:22.300470 7258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:22.301460 7258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:22.301510 7258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:22.301576 7258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:22.301600 7258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:22.311460 7258 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42353
I20260619 12:30:22.311488 7476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42353 every 8 connection(s)
I20260619 12:30:22.312470 7258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:30:22.315341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 7258
I20260619 12:30:22.315544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 6709
I20260619 12:30:22.323201 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34003
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43543
--webserver_port=45557
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43543,127.0.98.61:34171
--tserver_master_addrs=127.0.98.62:43543,127.0.98.61:34171,127.0.98.60:32971 with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:30:22 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872222, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 12:30:22 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872222, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 12:30:22 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872222, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 12:30:22.331573 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.314367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57785 (local address 127.0.98.60:32971)
0619 12:30:22.314616 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.314622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.315648 (+ 1026us) server_negotiation.cc:407] Connection header received
0619 12:30:22.317442 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.317447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.317512 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.317594 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.319598 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.320650 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.322572 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.322920 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.322986 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.326627 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.326631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.326633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.326665 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.328851 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.329561 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.329565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.329566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.329653 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.330086 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.330093 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.330098 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.330360 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.330453 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.331231 (+ 778us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.331381 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260619 12:30:22.332320 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.314416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54293 (local address 127.0.98.62:43543)
0619 12:30:22.314556 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.314561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.316423 (+ 1862us) server_negotiation.cc:407] Connection header received
0619 12:30:22.317513 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.317517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.317575 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.317627 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.319598 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.320441 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.322503 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.322841 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.322918 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.327776 (+ 4858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.327780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.327783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.327818 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.330033 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.330600 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.330604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.330606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.330723 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.331061 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.331064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.331066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.331307 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.331388 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.332048 (+ 660us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.332168 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 12:30:22.332383 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.316667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53679 (local address 127.0.98.61:34171)
0619 12:30:22.317286 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.317293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.318779 (+ 1486us) server_negotiation.cc:407] Connection header received
0619 12:30:22.318973 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.318979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.319039 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.319095 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.319800 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.320684 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.322491 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.322865 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.322948 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.328218 (+ 5270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.328223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.328229 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.328263 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.330123 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.330727 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.330733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.330738 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.330845 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.331192 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.331198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.331202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.331472 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.331569 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.331994 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.332158 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515}
I20260619 12:30:22.333348 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.314870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32971 (local address 127.0.98.1:57785)
0619 12:30:22.315483 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.315544 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.316813 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.317924 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.317933 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.318530 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:22.319423 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.319443 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.320789 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.320792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.322456 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.322465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.323140 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.323143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.323482 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.324351 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:22.324367 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:22.326451 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:22.328988 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.328997 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.329020 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.329436 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.329789 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.329795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.329798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.329976 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.330521 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:22.330529 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:22.330953 (+ 424us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.331841 (+ 888us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.332188 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":653,"spinlock_wait_cycles":61568,"thread_start_us":161,"threads_started":1}
I20260619 12:30:22.333365 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.316022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34171 (local address 127.0.98.1:53679)
0619 12:30:22.318679 (+ 2657us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.318696 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.318866 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.319267 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.319272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.319490 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:22.319640 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.319650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.320851 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.320857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.322358 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.322369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.323134 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.323141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.323453 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.324295 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:22.324322 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:22.328046 (+ 3724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:22.330283 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.330287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.330292 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.330594 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.330971 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.330976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.330980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.331077 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.331653 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:22.331659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:22.331781 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.332821 (+ 1040us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.332996 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2572,"mutex_wait_us":212,"spinlock_wait_cycles":17024,"thread_start_us":112,"threads_started":1}
I20260619 12:30:22.334229 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.314870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.1:54293)
0619 12:30:22.316329 (+ 1459us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.316346 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.316825 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.317924 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.317933 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.318530 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:22.319423 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.319443 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.320610 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.320613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.322350 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.322364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.323135 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.323139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.323511 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.324468 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:22.324487 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:22.327624 (+ 3137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:22.330165 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.330168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.330171 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.330458 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.330870 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.330873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.330875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.330956 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.331474 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:22.331476 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:22.331558 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.332922 (+ 1364us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.333062 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1339,"mutex_wait_us":321,"spinlock_wait_cycles":109952,"thread_start_us":708,"threads_started":1}
I20260619 12:30:22.334791 7477 heartbeater.cc:347] Connected to a master server at 127.0.98.60:32971
I20260619 12:30:22.335026 7477 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:22.335381 7478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43543
I20260619 12:30:22.335480 7478 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:22.335577 7477 heartbeater.cc:511] Master 127.0.98.60:32971 requested a full tablet report, sending...
I20260619 12:30:22.335704 7478 heartbeater.cc:511] Master 127.0.98.62:43543 requested a full tablet report, sending...
I20260619 12:30:22.336122 7479 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34171
I20260619 12:30:22.336194 7479 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:22.336354 7479 heartbeater.cc:511] Master 127.0.98.61:34171 requested a full tablet report, sending...
I20260619 12:30:22.336786 6513 ts_manager.cc:194] Re-registered known tserver with Master: 79139bcb3a2c449996505234878d5ed2 (127.0.98.1:42353)
I20260619 12:30:22.337184 7290 ts_manager.cc:194] Registered new tserver with Master: 79139bcb3a2c449996505234878d5ed2 (127.0.98.1:42353)
I20260619 12:30:22.337281 7114 ts_manager.cc:194] Registered new tserver with Master: 79139bcb3a2c449996505234878d5ed2 (127.0.98.1:42353)
I20260619 12:30:22.337560 6513 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54293
I20260619 12:30:22.436173 6972 heartbeater.cc:503] Master 127.0.98.62:43543 was elected leader, sending a full tablet report...
W20260619 12:30:22.443104 7485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:22.443374 7485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:22.443447 7485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:22.447489 7485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:22.447551 7485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:22.447654 7485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:30:22.453110 7485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45557
--tserver_master_addrs=127.0.98.62:43543,127.0.98.61:34171,127.0.98.60:32971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:22.454173 7485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:22.455279 7485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:22.465124 7491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:22.465301 7485 server_base.cc:1034] running on GCE node
W20260619 12:30:22.465178 7493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:22.465184 7490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:22.465951 7485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:22.466615 7485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:22.467816 7485 hybrid_clock.cc:648] HybridClock initialized: now 1781872222467799 us; error 31 us; skew 500 ppm
Jun 19 12:30:22 dist-test-slave-fg3h krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872222, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:22.470865 7485 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 12:30:22.471988 7485 webserver.cc:460] Webserver started at http://127.0.98.2:45557/ using document root <none> and password file <none>
I20260619 12:30:22.472573 7485 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:22.472684 7485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:22.476619 7485 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:22.479008 7500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:22.480396 7485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 12:30:22.480496 7485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "d437cfac39e64c909043a1c57af6668f"
format_stamp: "Formatted at 2026-06-19 12:30:21 on dist-test-slave-fg3h"
server_key: "d5716c29bd498381d019d2586ec08739"
server_key_iv: "dcc607c78a875be32877c99b4ee57708"
server_key_version: "encryptionkey@0"
I20260619 12:30:22.481031 7485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:22.503147 7485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:22.506448 7485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:22.506605 7485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:22.507182 7485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:22.507985 7485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:22.508033 7485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:22.508072 7485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:22.508093 7485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:22.517716 7485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34003
I20260619 12:30:22.517779 7613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34003 every 8 connection(s)
I20260619 12:30:22.518774 7485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:30:22.520664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 7485
I20260619 12:30:22.520865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 6843
I20260619 12:30:22.528741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34403
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43543
--webserver_port=40445
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43543,127.0.98.61:34171
--tserver_master_addrs=127.0.98.62:43543,127.0.98.61:34171,127.0.98.60:32971 with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:30:22 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872222, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 12:30:22 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872222, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 12:30:22 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872222, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 12:30:22.538643 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.528255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53725 (local address 127.0.98.61:34171)
0619 12:30:22.528400 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.528405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.528569 (+ 164us) server_negotiation.cc:407] Connection header received
0619 12:30:22.528751 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.528757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.528812 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.528868 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.529936 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.530706 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.532025 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.532344 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.532460 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.534789 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.534794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.534797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.534832 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.536748 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.537266 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.537271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.537273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.537364 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.537667 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.537671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.537673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.537884 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.537967 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.538318 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.538462 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:30:22.538988 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.521041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41767 (local address 127.0.98.60:32971)
0619 12:30:22.521559 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.521566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.524134 (+ 2568us) server_negotiation.cc:407] Connection header received
0619 12:30:22.524334 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.524340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.524407 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.524469 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.525872 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.526670 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.528661 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.529067 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.529201 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.533759 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.533767 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.533774 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.533814 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.536473 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.537150 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.537157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.537160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.537267 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.537656 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.537663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.537682 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.537986 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.538092 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.538543 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.538732 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415}
I20260619 12:30:22.539852 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.522799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34171 (local address 127.0.98.2:53725)
0619 12:30:22.528478 (+ 5679us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.528495 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.528640 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.529220 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.529226 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.529500 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:22.529758 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.529769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.530852 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.530855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.531894 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.531906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.532462 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.532465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.532554 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.533040 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:22.533056 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:22.534608 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:22.536856 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.536859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.536862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.537159 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.537475 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.537478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.537480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.537566 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.538022 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:22.538024 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:22.538098 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.538875 (+ 777us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.539017 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5608,"mutex_wait_us":78,"thread_start_us":106,"threads_started":1}
I20260619 12:30:22.540215 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.521597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41477 (local address 127.0.98.62:43543)
0619 12:30:22.522767 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.522775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.523323 (+ 548us) server_negotiation.cc:407] Connection header received
0619 12:30:22.524261 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.524264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.524320 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.524410 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.525824 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.526669 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.529763 (+ 3094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.530182 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.530300 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.533743 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.533747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.533750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.533780 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.535898 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.536649 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.536653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.536655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.536748 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.537212 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.537218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.537220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.537478 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.537568 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.539902 (+ 2334us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.540041 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1049}
I20260619 12:30:22.540705 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.521443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.2:41477)
0619 12:30:22.523165 (+ 1722us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.523211 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.524044 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.524590 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.524597 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.525018 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:22.525653 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.525682 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.528437 (+ 2755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.528440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.529615 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.529624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.530506 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.530510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.530647 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.531271 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:22.531286 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:22.533524 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:22.536046 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.536055 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.536084 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.536514 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.536865 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.536868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.536870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.537071 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.537622 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:22.537633 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:22.538020 (+ 387us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.538247 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.538613 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1515,"mutex_wait_us":6,"spinlock_wait_cycles":2816,"thread_start_us":687,"threads_started":1}
I20260619 12:30:22.541211 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.521443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32971 (local address 127.0.98.2:41767)
0619 12:30:22.524025 (+ 2582us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.524044 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.524213 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.524645 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.524651 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.525072 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:22.525656 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.525689 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.526847 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.526854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.528511 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.528520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.529408 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.529412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.529695 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.530483 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:22.530505 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:22.533545 (+ 3040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:22.536632 (+ 3087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.536637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.536641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.536976 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.537411 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.537416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.537420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.537527 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.538166 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:22.538172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:22.538296 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.539689 (+ 1393us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.540582 (+ 893us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2458,"mutex_wait_us":56,"spinlock_wait_cycles":30208,"thread_start_us":178,"threads_started":1}
I20260619 12:30:22.541319 7616 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34171
I20260619 12:30:22.541587 7616 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:22.542042 7614 heartbeater.cc:347] Connected to a master server at 127.0.98.60:32971
I20260619 12:30:22.542135 7614 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:22.542275 7616 heartbeater.cc:511] Master 127.0.98.61:34171 requested a full tablet report, sending...
I20260619 12:30:22.542303 7614 heartbeater.cc:511] Master 127.0.98.60:32971 requested a full tablet report, sending...
I20260619 12:30:22.542892 7615 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43543
I20260619 12:30:22.542979 7615 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:22.543191 7615 heartbeater.cc:511] Master 127.0.98.62:43543 requested a full tablet report, sending...
I20260619 12:30:22.543681 7290 ts_manager.cc:194] Registered new tserver with Master: d437cfac39e64c909043a1c57af6668f (127.0.98.2:34003)
I20260619 12:30:22.543874 6513 ts_manager.cc:194] Re-registered known tserver with Master: d437cfac39e64c909043a1c57af6668f (127.0.98.2:34003)
I20260619 12:30:22.544446 7114 ts_manager.cc:194] Registered new tserver with Master: d437cfac39e64c909043a1c57af6668f (127.0.98.2:34003)
I20260619 12:30:22.544603 6513 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41477
W20260619 12:30:22.653856 7621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:22.654117 7621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:22.654160 7621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:22.658254 7621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:22.658308 7621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:22.658394 7621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:30:22.663396 7625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.659266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32971 (local address 127.0.0.1:38012)
0619 12:30:22.659516 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.659529 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.659637 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.659965 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.659968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.659986 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:22.660081 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.660085 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.661181 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.661185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.662132 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.662143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.663045 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.663048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.663119 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.663141 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.663215 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.663276 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20260619 12:30:22.663394 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.659309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38012 (local address 127.0.98.60:32971)
0619 12:30:22.659454 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.659459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.659615 (+ 156us) server_negotiation.cc:407] Connection header received
0619 12:30:22.659762 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.659765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.659829 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.659930 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:22.660192 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.661027 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.662264 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.662923 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.663030 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.663171 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.663206 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.663271 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 12:30:22.664034 7300 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 12:30:22.664466 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5163b08470e840a98fb193929dc7ad25. Latest consensus state: current_term: 1 leader_uuid: "5163b08470e840a98fb193929dc7ad25" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: true } } }
I20260619 12:30:22.664624 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [sys.catalog]: This master's current role is: LEARNER
I20260619 12:30:22.665309 7621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40445
--tserver_master_addrs=127.0.98.62:43543,127.0.98.61:34171,127.0.98.60:32971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:22.666285 7621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:22.667222 7621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:22.676579 7628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:22.676595 7627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:22.676528 7630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:22.676913 7621 server_base.cc:1034] running on GCE node
I20260619 12:30:22.677407 7621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:22.678002 7621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:22.679189 7621 hybrid_clock.cc:648] HybridClock initialized: now 1781872222679162 us; error 46 us; skew 500 ppm
Jun 19 12:30:22 dist-test-slave-fg3h krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872222, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:22.682045 7621 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 12:30:22.682951 7621 webserver.cc:460] Webserver started at http://127.0.98.3:40445/ using document root <none> and password file <none>
I20260619 12:30:22.683523 7621 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:22.683626 7621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:22.687461 7621 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:30:22.689605 7637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:22.690909 7621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:30:22.691049 7621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "ecb3e17ffe404886b679448d181926dc"
format_stamp: "Formatted at 2026-06-19 12:30:21 on dist-test-slave-fg3h"
server_key: "ab3b59e26c969bee91e7b641c3d39870"
server_key_iv: "e6ae32b17b001bba78af99949333310a"
server_key_version: "encryptionkey@0"
I20260619 12:30:22.691664 7621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:22.704442 7621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:22.707772 7621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:22.707974 7621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:22.708549 7621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:22.709494 7621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:22.709551 7621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:22.709614 7621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:22.709645 7621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:22.718199 7621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34403
I20260619 12:30:22.718390 7750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34403 every 8 connection(s)
I20260619 12:30:22.719218 7621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:30:22.727396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 7621
Jun 19 12:30:22 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872222, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 12:30:22 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872222, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 12:30:22 dist-test-slave-fg3h krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872222, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 12:30:22.739358 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.721024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46913 (local address 127.0.98.61:34171)
0619 12:30:22.721204 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.721209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.722620 (+ 1411us) server_negotiation.cc:407] Connection header received
0619 12:30:22.723635 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.723639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.723695 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.723823 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.725039 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.725690 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.727690 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.728052 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.728122 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.732370 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.732376 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.732382 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.732417 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.735569 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.736309 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.736313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.736315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.736397 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.737121 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.737126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.737129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.737378 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.737468 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.739100 (+ 1632us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.739212 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 12:30:22.739566 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.721520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34171 (local address 127.0.98.3:46913)
0619 12:30:22.722507 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.722521 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.723313 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.723941 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.723944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.724388 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:22.724879 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.724890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.725845 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.725849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.727445 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.727573 (+ 128us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.728290 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.728294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.728394 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.729449 (+ 1055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:22.729464 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:22.732219 (+ 2755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:22.735952 (+ 3733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.735955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.735957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.736212 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.736497 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.736499 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.736501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.736592 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.737725 (+ 1133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:22.737731 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:22.737897 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.738046 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.738423 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":287,"spinlock_wait_cycles":79104,"thread_start_us":701,"threads_started":1}
I20260619 12:30:22.740418 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.722838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55431 (local address 127.0.98.62:43543)
0619 12:30:22.722974 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.722977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.723082 (+ 105us) server_negotiation.cc:407] Connection header received
0619 12:30:22.723526 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.723529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.723584 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.723697 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.725013 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.725892 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.727317 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.727642 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.727721 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.731757 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.731763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.731766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.731795 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.733900 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.735550 (+ 1650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.735557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.735561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.735661 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.736161 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.736168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.736174 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.736457 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.736549 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.740088 (+ 3539us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.740223 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:30:22.740455 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.721520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32971 (local address 127.0.98.3:59691)
0619 12:30:22.722078 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.722123 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.723313 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.723888 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.723898 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.724387 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:22.724882 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.724890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.726003 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.726007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.727446 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.727457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.728248 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.728252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.728362 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.729000 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:22.729019 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:22.731852 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:22.734450 (+ 2598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.734456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.734471 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.734788 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.736901 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.736905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.736906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.737001 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.739518 (+ 2517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:22.739521 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:22.739631 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.739735 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.739889 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":286,"spinlock_wait_cycles":13952,"thread_start_us":175,"threads_started":1}
I20260619 12:30:22.740936 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.721030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59691 (local address 127.0.98.60:32971)
0619 12:30:22.721297 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.721300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.722239 (+ 939us) server_negotiation.cc:407] Connection header received
0619 12:30:22.723557 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.723561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.723627 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.723688 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.725051 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.725892 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.727597 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.727985 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.728066 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.732125 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.732131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.732138 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.732167 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.734329 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.736694 (+ 2365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.736700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.736705 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.736786 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.737375 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.737378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.737380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.737587 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.737677 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.740681 (+ 3004us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.740796 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260619 12:30:22.741204 7753 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34171
I20260619 12:30:22.741490 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.729694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38018 (local address 127.0.98.60:32971)
0619 12:30:22.730616 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.730624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.730877 (+ 253us) server_negotiation.cc:407] Connection header received
0619 12:30:22.730922 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.730925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.731047 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.731661 (+ 614us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.732243 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.733144 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.734680 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.735112 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.735218 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.737049 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.737052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.737055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.737078 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.739313 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.739852 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.739856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.739858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.739944 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.740267 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.740270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.740272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.740490 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.740633 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.741211 (+ 578us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.741333 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":89,"threads_started":1}
I20260619 12:30:22.741506 7753 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:22.741901 7751 heartbeater.cc:347] Connected to a master server at 127.0.98.60:32971
I20260619 12:30:22.741978 7751 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:22.742069 7753 heartbeater.cc:511] Master 127.0.98.61:34171 requested a full tablet report, sending...
I20260619 12:30:22.742147 7751 heartbeater.cc:511] Master 127.0.98.60:32971 requested a full tablet report, sending...
I20260619 12:30:22.743301 7114 ts_manager.cc:194] Registered new tserver with Master: ecb3e17ffe404886b679448d181926dc (127.0.98.3:34403)
I20260619 12:30:22.743341 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.722741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.3:55431)
0619 12:30:22.723006 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.723021 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.723335 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.723887 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.723895 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.724317 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:22.724863 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.724877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.726057 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.726060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.727164 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.727177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.727952 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.727957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.728151 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.728821 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:22.728844 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:22.731575 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:22.735129 (+ 3554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.735136 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.735141 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.735423 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.735793 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:22.735795 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:22.735797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:22.736024 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:22.736651 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:22.736661 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:22.737019 (+ 358us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.741346 (+ 4327us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.743102 (+ 1756us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":195,"spinlock_wait_cycles":23296,"thread_start_us":79,"threads_started":1}
I20260619 12:30:22.743480 7290 ts_manager.cc:194] Registered new tserver with Master: ecb3e17ffe404886b679448d181926dc (127.0.98.3:34403)
I20260619 12:30:22.744745 7752 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43543
I20260619 12:30:22.744833 7752 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:22.745008 7752 heartbeater.cc:511] Master 127.0.98.62:43543 requested a full tablet report, sending...
I20260619 12:30:22.745823 6513 ts_manager.cc:194] Re-registered known tserver with Master: ecb3e17ffe404886b679448d181926dc (127.0.98.3:34403)
I20260619 12:30:22.746668 6513 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55431
I20260619 12:30:22.747162 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.730165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35702 (local address 127.0.98.62:43543)
0619 12:30:22.730600 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.730613 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.730848 (+ 235us) server_negotiation.cc:407] Connection header received
0619 12:30:22.731035 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.731038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.731092 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.731164 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.734126 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.734943 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.736140 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.736532 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.736606 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.738701 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.738705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.738708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.738734 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.744794 (+ 6060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.745705 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.745708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.745710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.745803 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.746136 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.746139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.746141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.746403 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.746529 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.746872 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.747006 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":362,"thread_start_us":80,"threads_started":1}
I20260619 12:30:22.747836 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.730731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46396 (local address 127.0.98.61:34171)
0619 12:30:22.731916 (+ 1185us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.731920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.732861 (+ 941us) server_negotiation.cc:407] Connection header received
0619 12:30:22.734022 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.734026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.734077 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.734139 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.737340 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.738496 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.741799 (+ 3303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.742137 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.742214 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.743965 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.743968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.743971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.743995 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.745842 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.746291 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.746294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.746296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.746382 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.746658 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.746661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.746663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.746899 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.747016 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.747614 (+ 598us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.747718 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1100,"thread_start_us":77,"threads_started":1}
I20260619 12:30:22.750005 6513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:30:22.753005 6513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:30:22.755956 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5163b08470e840a98fb193929dc7ad25" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: true } } }
I20260619 12:30:22.756095 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:22.759840 7769 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25: attempting to promote NON_VOTER 6520e328cbfb4ba8aefb2874c02d3dec to VOTER
I20260619 12:30:22.760179 7769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } }
I20260619 12:30:22.761416 7123 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 1 FOLLOWER]: Refusing update from remote peer 5163b08470e840a98fb193929dc7ad25: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 12:30:22.761879 7770 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 12:30:22.762876 7300 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 1 LEARNER]: Refusing update from remote peer 5163b08470e840a98fb193929dc7ad25: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 12:30:22.763269 7770 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 12:30:22.763602 7074 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6520e328cbfb4ba8aefb2874c02d3dec (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } } }
I20260619 12:30:22.764287 7123 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6520e328cbfb4ba8aefb2874c02d3dec (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } } }
I20260619 12:30:22.765210 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5163b08470e840a98fb193929dc7ad25" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } } }
I20260619 12:30:22.765321 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:22.765743 7074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5163b08470e840a98fb193929dc7ad25" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } } }
I20260619 12:30:22.765858 7074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:22.766330 7778 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5163b08470e840a98fb193929dc7ad25" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } } }
I20260619 12:30:22.766474 7778 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:30:22.766795 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5163b08470e840a98fb193929dc7ad25" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } } }
I20260619 12:30:22.766922 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:22.767210 7300 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6520e328cbfb4ba8aefb2874c02d3dec (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } } }
I20260619 12:30:22.770082 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5163b08470e840a98fb193929dc7ad25" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } } }
I20260619 12:30:22.770210 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:30:22.774294 7625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.768326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34003 (local address 127.0.0.1:45314)
0619 12:30:22.768425 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.768442 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.768524 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.770047 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.770050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.770071 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:22.770199 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.770208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.771563 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.771567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.772735 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.772745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.773934 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.773938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.774027 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.774054 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.774100 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.774165 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 12:30:22.775142 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.768660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45314 (local address 127.0.98.2:34003)
0619 12:30:22.769520 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.769533 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.769560 (+ 27us) server_negotiation.cc:407] Connection header received
0619 12:30:22.769640 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.769645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.769899 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.770079 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:22.770353 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.771413 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.772887 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.773798 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.774707 (+ 909us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.774862 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.774913 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.775006 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":609,"threads_started":1}
I20260619 12:30:22.775856 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.769929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34403 (local address 127.0.0.1:37700)
0619 12:30:22.771321 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.771338 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.771444 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.771896 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.771899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.771924 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:22.772019 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.772025 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.773340 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.773343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.774530 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.774539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.775450 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.775454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.775544 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.775580 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.775654 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.775720 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1313,"mutex_wait_us":1,"thread_start_us":108,"threads_started":1}
I20260619 12:30:22.776057 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.770765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37700 (local address 127.0.98.3:34403)
0619 12:30:22.771171 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.771186 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.771443 (+ 257us) server_negotiation.cc:407] Connection header received
0619 12:30:22.771563 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.771568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.771765 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.771933 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:22.772124 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.773222 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.774656 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.775307 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.775438 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.775578 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.775676 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.775764 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":113,"threads_started":1}
I20260619 12:30:22.776906 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.770240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42353 (local address 127.0.0.1:42114)
0619 12:30:22.771613 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.771623 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.771729 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.772183 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.772186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.772202 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:22.772317 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.772326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.773480 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.773484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.774593 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.774603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.775835 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.775839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.775937 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.775965 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.776038 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.776098 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1327,"thread_start_us":89,"threads_started":1}
I20260619 12:30:22.777415 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.770748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42114 (local address 127.0.98.1:42353)
0619 12:30:22.771344 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.771363 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.771714 (+ 351us) server_negotiation.cc:407] Connection header received
0619 12:30:22.771850 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.771855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.772058 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.772212 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:22.772446 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.773363 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.774724 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.775528 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.775636 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.775767 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.777185 (+ 1418us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.777275 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":74,"threads_started":1}
I20260619 12:30:22.778368 7548 tablet_service.cc:1467] Processing CreateTablet for tablet 7570a1cba1aa4936a015ccbbc46f9dac (DEFAULT_TABLE table=test-table [id=bfdff6f0f01645e6bc26bdac9bef2327]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:22.778834 7685 tablet_service.cc:1467] Processing CreateTablet for tablet 7570a1cba1aa4936a015ccbbc46f9dac (DEFAULT_TABLE table=test-table [id=bfdff6f0f01645e6bc26bdac9bef2327]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:22.779371 7548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7570a1cba1aa4936a015ccbbc46f9dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:22.779901 7685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7570a1cba1aa4936a015ccbbc46f9dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:22.780129 7411 tablet_service.cc:1467] Processing CreateTablet for tablet 7570a1cba1aa4936a015ccbbc46f9dac (DEFAULT_TABLE table=test-table [id=bfdff6f0f01645e6bc26bdac9bef2327]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:22.781164 7411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7570a1cba1aa4936a015ccbbc46f9dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:22.788116 7786 tablet_bootstrap.cc:492] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc: Bootstrap starting.
I20260619 12:30:22.789119 7787 tablet_bootstrap.cc:492] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2: Bootstrap starting.
I20260619 12:30:22.789490 7785 tablet_bootstrap.cc:492] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f: Bootstrap starting.
I20260619 12:30:22.790401 7786 tablet_bootstrap.cc:654] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:22.790889 7787 tablet_bootstrap.cc:654] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:22.791249 7786 log.cc:824] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:22.791549 7785 tablet_bootstrap.cc:654] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:22.791656 7787 log.cc:824] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:22.792306 7785 log.cc:824] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:22.793818 7786 tablet_bootstrap.cc:492] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc: No bootstrap required, opened a new log
I20260619 12:30:22.793838 7787 tablet_bootstrap.cc:492] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2: No bootstrap required, opened a new log
I20260619 12:30:22.794023 7787 ts_tablet_manager.cc:1397] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:30:22.794025 7786 ts_tablet_manager.cc:1397] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:30:22.794269 7785 tablet_bootstrap.cc:492] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f: No bootstrap required, opened a new log
I20260619 12:30:22.794507 7785 ts_tablet_manager.cc:1397] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 12:30:22.798830 7787 raft_consensus.cc:348] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } } peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } }
I20260619 12:30:22.799094 7787 raft_consensus.cc:374] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:22.799150 7787 raft_consensus.cc:729] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79139bcb3a2c449996505234878d5ed2, State: Initialized, Role: FOLLOWER
I20260619 12:30:22.799105 7786 raft_consensus.cc:348] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } } peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } }
I20260619 12:30:22.799343 7786 raft_consensus.cc:374] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:22.799391 7786 raft_consensus.cc:729] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb3e17ffe404886b679448d181926dc, State: Initialized, Role: FOLLOWER
I20260619 12:30:22.799496 7785 raft_consensus.cc:348] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } } peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } }
I20260619 12:30:22.799623 7787 consensus_queue.cc:260] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } } peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } }
I20260619 12:30:22.799726 7785 raft_consensus.cc:374] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:22.799779 7785 raft_consensus.cc:729] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d437cfac39e64c909043a1c57af6668f, State: Initialized, Role: FOLLOWER
I20260619 12:30:22.799794 7786 consensus_queue.cc:260] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } } peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } }
I20260619 12:30:22.800160 7785 consensus_queue.cc:260] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } } peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } }
I20260619 12:30:22.800643 7787 ts_tablet_manager.cc:1428] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 12:30:22.800977 7785 ts_tablet_manager.cc:1428] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 12:30:22.801215 7752 heartbeater.cc:503] Master 127.0.98.62:43543 was elected leader, sending a full tablet report...
I20260619 12:30:22.801589 7615 heartbeater.cc:503] Master 127.0.98.62:43543 was elected leader, sending a full tablet report...
I20260619 12:30:22.801857 7478 heartbeater.cc:503] Master 127.0.98.62:43543 was elected leader, sending a full tablet report...
I20260619 12:30:22.804957 7786 ts_tablet_manager.cc:1428] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
W20260619 12:30:22.813994 7480 tablet.cc:2369] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:30:22.877666 7791 raft_consensus.cc:482] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:30:22.877955 7791 raft_consensus.cc:504] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } } peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } }
I20260619 12:30:22.879091 7791 leader_election.cc:288] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d437cfac39e64c909043a1c57af6668f (127.0.98.2:34003), ecb3e17ffe404886b679448d181926dc (127.0.98.3:34403)
I20260619 12:30:22.883054 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.879505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47791 (local address 127.0.98.3:34403)
0619 12:30:22.879637 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.879642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.879819 (+ 177us) server_negotiation.cc:407] Connection header received
0619 12:30:22.879973 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.879979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.880036 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.880094 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:22.880416 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.880971 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.882090 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.882475 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.882555 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.882646 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.882828 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.882917 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:30:22.883112 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.879527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38075 (local address 127.0.98.2:34003)
0619 12:30:22.879727 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.879734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.879814 (+ 80us) server_negotiation.cc:407] Connection header received
0619 12:30:22.880007 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.880010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.880051 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.880144 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:22.880467 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.880974 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.882099 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.882502 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.882630 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.882692 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.882916 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.882997 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260619 12:30:22.883076 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.879399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34403 (local address 127.0.98.1:47791)
0619 12:30:22.879699 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.879716 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.879851 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.880129 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.880135 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.880152 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:22.880303 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.880308 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.881137 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.881141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.881961 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.881970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.882594 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.882601 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.882693 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.882716 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.882797 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.882887 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":60,"thread_start_us":87,"threads_started":1}
I20260619 12:30:22.883096 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.879399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34003 (local address 127.0.98.1:38075)
0619 12:30:22.879724 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:22.879735 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:22.879871 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:22.880179 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:22.880183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:22.880202 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:22.880313 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.880317 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.881122 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.881127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.881954 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:22.881962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.882646 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.882650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:22.882754 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.882793 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 12:30:22.882884 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 12:30:22.882945 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":39,"spinlock_wait_cycles":1280,"thread_start_us":76,"threads_started":1}
I20260619 12:30:22.883960 7568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7570a1cba1aa4936a015ccbbc46f9dac" candidate_uuid: "79139bcb3a2c449996505234878d5ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d437cfac39e64c909043a1c57af6668f" is_pre_election: true
I20260619 12:30:22.883998 7705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7570a1cba1aa4936a015ccbbc46f9dac" candidate_uuid: "79139bcb3a2c449996505234878d5ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb3e17ffe404886b679448d181926dc" is_pre_election: true
I20260619 12:30:22.884274 7568 raft_consensus.cc:2454] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79139bcb3a2c449996505234878d5ed2 in term 0.
I20260619 12:30:22.884367 7705 raft_consensus.cc:2454] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79139bcb3a2c449996505234878d5ed2 in term 0.
I20260619 12:30:22.884753 7364 leader_election.cc:302] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79139bcb3a2c449996505234878d5ed2, d437cfac39e64c909043a1c57af6668f; no voters:
I20260619 12:30:22.885057 7791 raft_consensus.cc:2781] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:30:22.885190 7791 raft_consensus.cc:482] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:30:22.885459 7791 raft_consensus.cc:3037] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:22.886554 7791 raft_consensus.cc:504] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } } peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } }
I20260619 12:30:22.886933 7791 leader_election.cc:288] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [CANDIDATE]: Term 1 election: Requested vote from peers d437cfac39e64c909043a1c57af6668f (127.0.98.2:34003), ecb3e17ffe404886b679448d181926dc (127.0.98.3:34403)
I20260619 12:30:22.887253 7568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7570a1cba1aa4936a015ccbbc46f9dac" candidate_uuid: "79139bcb3a2c449996505234878d5ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d437cfac39e64c909043a1c57af6668f"
I20260619 12:30:22.887310 7705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7570a1cba1aa4936a015ccbbc46f9dac" candidate_uuid: "79139bcb3a2c449996505234878d5ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb3e17ffe404886b679448d181926dc"
I20260619 12:30:22.887380 7568 raft_consensus.cc:3037] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:22.887418 7705 raft_consensus.cc:3037] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:22.888639 7705 raft_consensus.cc:2454] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79139bcb3a2c449996505234878d5ed2 in term 1.
I20260619 12:30:22.888639 7568 raft_consensus.cc:2454] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79139bcb3a2c449996505234878d5ed2 in term 1.
I20260619 12:30:22.888943 7367 leader_election.cc:302] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79139bcb3a2c449996505234878d5ed2, ecb3e17ffe404886b679448d181926dc; no voters:
I20260619 12:30:22.889145 7791 raft_consensus.cc:2781] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:22.889384 7791 raft_consensus.cc:686] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [term 1 LEADER]: Becoming Leader. State: Replica: 79139bcb3a2c449996505234878d5ed2, State: Running, Role: LEADER
I20260619 12:30:22.889725 7791 consensus_queue.cc:237] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } } peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } }
I20260619 12:30:22.893517 6511 catalog_manager.cc:5495] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79139bcb3a2c449996505234878d5ed2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "79139bcb3a2c449996505234878d5ed2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } health_report { overall_health: HEALTHY } } }
I20260619 12:30:22.928364 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.924708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42122 (local address 127.0.98.1:42353)
0619 12:30:22.924836 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.924840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.924853 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:30:22.925054 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.925056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.925135 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.925226 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:30:22.925459 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.925971 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.926796 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.927051 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.927157 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.927445 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:30:22.927961 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:30:22.928058 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.928186 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.928247 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:30:22.934448 7705 raft_consensus.cc:1261] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc [term 1 FOLLOWER]: Refusing update from remote peer 79139bcb3a2c449996505234878d5ed2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:22.934458 7568 raft_consensus.cc:1261] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f [term 1 FOLLOWER]: Refusing update from remote peer 79139bcb3a2c449996505234878d5ed2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:22.935176 7791 consensus_queue.cc:1035] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:22.935386 7796 consensus_queue.cc:1035] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:22.943640 7800 mvcc.cc:205] Tried to move back new op lower bound from 7298548625132396544 to 7298548624959082496. Current Snapshot: MvccSnapshot[applied={T|T < 7298548625132396544}]
I20260619 12:30:22.948884 7801 mvcc.cc:205] Tried to move back new op lower bound from 7298548625132396544 to 7298548624959082496. Current Snapshot: MvccSnapshot[applied={T|T < 7298548625132396544}]
I20260619 12:30:22.952654 6512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:22.952824 6512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:22.957587 6512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5163b08470e840a98fb193929dc7ad25: Sending DeleteTablet for 3 replicas of tablet 7570a1cba1aa4936a015ccbbc46f9dac
I20260619 12:30:22.958259 7548 tablet_service.cc:1514] Processing DeleteTablet for tablet 7570a1cba1aa4936a015ccbbc46f9dac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45314
I20260619 12:30:22.958560 7411 tablet_service.cc:1514] Processing DeleteTablet for tablet 7570a1cba1aa4936a015ccbbc46f9dac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42114
I20260619 12:30:22.958773 7809 tablet_replica.cc:330] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f: stopping tablet replica
I20260619 12:30:22.959031 7810 tablet_replica.cc:330] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2: stopping tablet replica
I20260619 12:30:22.959033 7685 tablet_service.cc:1514] Processing DeleteTablet for tablet 7570a1cba1aa4936a015ccbbc46f9dac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37700
I20260619 12:30:22.959089 7809 raft_consensus.cc:2229] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:22.959391 7809 raft_consensus.cc:2258] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:22.959414 7810 raft_consensus.cc:2229] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:30:22.959833 7810 raft_consensus.cc:2258] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:22.960343 7811 tablet_replica.cc:330] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc: stopping tablet replica
I20260619 12:30:22.960615 7811 raft_consensus.cc:2229] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:22.960897 7811 raft_consensus.cc:2258] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:22.961634 7810 ts_tablet_manager.cc:1905] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:22.962124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 6481
I20260619 12:30:22.963181 7809 ts_tablet_manager.cc:1905] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:22.964680 7810 ts_tablet_manager.cc:1918] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:30:22.964849 7810 log.cc:1196] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-0/wal/wals/7570a1cba1aa4936a015ccbbc46f9dac
I20260619 12:30:22.965262 7810 ts_tablet_manager.cc:1939] T 7570a1cba1aa4936a015ccbbc46f9dac P 79139bcb3a2c449996505234878d5ed2: Deleting consensus metadata
I20260619 12:30:22.967051 7809 ts_tablet_manager.cc:1918] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:30:22.967154 7809 log.cc:1196] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/7570a1cba1aa4936a015ccbbc46f9dac
I20260619 12:30:22.967475 7809 ts_tablet_manager.cc:1939] T 7570a1cba1aa4936a015ccbbc46f9dac P d437cfac39e64c909043a1c57af6668f: Deleting consensus metadata
W20260619 12:30:22.970223 7366 connection.cc:535] server connection from 127.0.0.1:42114 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 12:30:22.970490 7502 connection.cc:535] server connection from 127.0.0.1:45314 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 12:30:22.972265 7811 ts_tablet_manager.cc:1905] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:22.974635 7811 ts_tablet_manager.cc:1918] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:30:22.974710 7811 log.cc:1196] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/7570a1cba1aa4936a015ccbbc46f9dac
I20260619 12:30:22.974933 7811 ts_tablet_manager.cc:1939] T 7570a1cba1aa4936a015ccbbc46f9dac P ecb3e17ffe404886b679448d181926dc: Deleting consensus metadata
W20260619 12:30:22.975906 7638 connection.cc:441] server connection from 127.0.0.1:37700 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37700 (request call id 1) could send its response
I20260619 12:30:22.977015 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.961182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.0.98.61:34171)
0619 12:30:22.961303 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.961309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.966718 (+ 5409us) server_negotiation.cc:407] Connection header received
0619 12:30:22.966856 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.966860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.966912 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.966989 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.968169 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.968994 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.970376 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.970734 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.970831 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.973521 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.973525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.973528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.973554 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.975332 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.975895 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.975898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.975901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.975981 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.976253 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.976256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.976257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.976436 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.976535 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.976768 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.976866 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 12:30:22.987342 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.961291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.0.98.60:32971)
0619 12:30:22.961384 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.961389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.977168 (+ 15779us) server_negotiation.cc:407] Connection header received
0619 12:30:22.977337 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.977341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.977400 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.977483 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.978251 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.979006 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.980283 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.980679 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.980769 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.982482 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.982486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.982490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.982516 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.984980 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.985600 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.985603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.985605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.985691 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.986042 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.986048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.986050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.986314 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.986448 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.987010 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.987137 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":26,"spinlock_wait_cycles":6528}
I20260619 12:30:22.988003 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.979166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38024 (local address 127.0.98.60:32971)
0619 12:30:22.979274 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.979278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.979291 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:30:22.979324 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.979327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.979367 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.979442 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.980296 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.981080 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.982294 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.982642 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.982713 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.984340 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.984344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.984347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.984372 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.986299 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.986758 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.986762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.986764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.986851 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.987138 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.987141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.987143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.987333 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.987426 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.987756 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.987872 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":36}
I20260619 12:30:22.988756 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:22.979000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46420 (local address 127.0.98.61:34171)
0619 12:30:22.979120 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:22.979124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:22.979521 (+ 397us) server_negotiation.cc:407] Connection header received
0619 12:30:22.979674 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:22.979677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:22.979715 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:22.979775 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:22.980626 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.981404 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.982639 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:22.983040 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:22.983120 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:22.985061 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:22.985065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:22.985069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:22.985149 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:22.986853 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.987386 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.987392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.987393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.987481 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:22.987779 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:22.987782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:22.987784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:22.987986 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:22.988100 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:22.988552 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 12:30:22.988652 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:30:23.951156 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:23.950519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.1:42463)
0619 12:30:23.950923 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:23.951024 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":93,"threads_started":1}
W20260619 12:30:23.951526 7478 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43543 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
I20260619 12:30:23.952893 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:23.952539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.2:53933)
0619 12:30:23.952753 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:23.952820 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
W20260619 12:30:23.953224 7615 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43543 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
I20260619 12:30:23.953797 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:23.953427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.3:37661)
0619 12:30:23.953644 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:23.953713 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
W20260619 12:30:23.954246 7752 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43543 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
I20260619 12:30:24.457057 7822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5163b08470e840a98fb193929dc7ad25)
I20260619 12:30:24.457363 7822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } }
I20260619 12:30:24.457419 7823 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5163b08470e840a98fb193929dc7ad25)
I20260619 12:30:24.457597 7823 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } }
I20260619 12:30:24.458367 7822 leader_election.cc:288] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5163b08470e840a98fb193929dc7ad25 (127.0.98.62:43543), d3a7477e1f1e4b52bdf624890ad3c070 (127.0.98.61:34171)
I20260619 12:30:24.458621 7823 leader_election.cc:288] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5163b08470e840a98fb193929dc7ad25 (127.0.98.62:43543), 6520e328cbfb4ba8aefb2874c02d3dec (127.0.98.60:32971)
I20260619 12:30:24.459046 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:24.458550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:36134)
0619 12:30:24.458859 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:24.458941 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":58,"threads_started":1}
I20260619 12:30:24.459249 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:24.458735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:36138)
0619 12:30:24.459079 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:24.459155 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":58,"threads_started":1}
W20260619 12:30:24.459547 7100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
W20260619 12:30:24.459677 7284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
I20260619 12:30:24.460629 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:24.460337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:36144)
0619 12:30:24.460508 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:24.460562 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260619 12:30:24.460778 7284 leader_election.cc:334] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5163b08470e840a98fb193929dc7ad25 (127.0.98.62:43543): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
I20260619 12:30:24.461035 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:24.460536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:36146)
0619 12:30:24.460665 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:24.460957 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260619 12:30:24.461177 7100 leader_election.cc:334] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5163b08470e840a98fb193929dc7ad25 (127.0.98.62:43543): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
I20260619 12:30:24.463341 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:24.458889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34171 (local address 127.0.0.1:46422)
0619 12:30:24.459179 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:24.459192 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:24.459298 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:24.459694 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:24.459701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:24.459721 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:24.459843 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:24.459853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:24.460966 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:24.460970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:24.462268 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:24.462278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:24.462915 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:24.462919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:24.463023 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:24.463042 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 12:30:24.463131 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 12:30:24.463201 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":49,"threads_started":1}
I20260619 12:30:24.463331 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:24.458759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32971 (local address 127.0.0.1:38040)
0619 12:30:24.459286 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:24.459373 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:24.459469 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:24.459853 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:24.459856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:24.459874 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:24.459970 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:24.459975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:24.460970 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:24.460973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:24.462268 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:24.462278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:24.462935 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:24.462938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:24.463022 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:24.463046 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 12:30:24.463134 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 12:30:24.463200 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":55,"threads_started":1}
I20260619 12:30:24.463331 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:24.459139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46422 (local address 127.0.98.61:34171)
0619 12:30:24.459462 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:24.459467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:24.459480 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:30:24.459521 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:24.459525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:24.459574 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:24.459958 (+ 384us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:24.460074 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:24.460817 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:24.462419 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:24.462799 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:24.462919 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:24.462989 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:24.463145 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 12:30:24.463200 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":271,"thread_start_us":66,"threads_started":1}
I20260619 12:30:24.463354 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:24.459317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.0.98.60:32971)
0619 12:30:24.459535 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:24.459542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:24.459558 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:30:24.459601 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:24.459604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:24.459645 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:24.459727 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:24.460115 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:24.460874 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:24.462390 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:24.462796 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:24.462935 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:24.462992 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:24.463146 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 12:30:24.463201 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":61,"threads_started":1}
I20260619 12:30:24.463892 7123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" is_pre_election: true
I20260619 12:30:24.464159 7123 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6520e328cbfb4ba8aefb2874c02d3dec in term 1.
I20260619 12:30:24.464175 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" is_pre_election: true
I20260619 12:30:24.464327 7300 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3a7477e1f1e4b52bdf624890ad3c070 in term 1.
I20260619 12:30:24.464519 7284 leader_election.cc:302] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6520e328cbfb4ba8aefb2874c02d3dec, d3a7477e1f1e4b52bdf624890ad3c070; no voters: 5163b08470e840a98fb193929dc7ad25
I20260619 12:30:24.464633 7101 leader_election.cc:302] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6520e328cbfb4ba8aefb2874c02d3dec, d3a7477e1f1e4b52bdf624890ad3c070; no voters: 5163b08470e840a98fb193929dc7ad25
I20260619 12:30:24.464749 7822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 12:30:24.464812 7822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5163b08470e840a98fb193929dc7ad25)
I20260619 12:30:24.464865 7822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:30:24.464895 7823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 12:30:24.464952 7823 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5163b08470e840a98fb193929dc7ad25)
I20260619 12:30:24.464989 7823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:30:24.465983 7822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } }
I20260619 12:30:24.466005 7823 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } }
I20260619 12:30:24.466353 7822 leader_election.cc:288] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [CANDIDATE]: Term 2 election: Requested vote from peers 5163b08470e840a98fb193929dc7ad25 (127.0.98.62:43543), d3a7477e1f1e4b52bdf624890ad3c070 (127.0.98.61:34171)
I20260619 12:30:24.466403 7823 leader_election.cc:288] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [CANDIDATE]: Term 2 election: Requested vote from peers 5163b08470e840a98fb193929dc7ad25 (127.0.98.62:43543), 6520e328cbfb4ba8aefb2874c02d3dec (127.0.98.60:32971)
I20260619 12:30:24.466728 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:24.466493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:36154)
0619 12:30:24.466604 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:24.466660 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 12:30:24.466795 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:24.466507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:36162)
0619 12:30:24.466666 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:24.466705 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260619 12:30:24.466779 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6520e328cbfb4ba8aefb2874c02d3dec"
I20260619 12:30:24.466851 7123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d3a7477e1f1e4b52bdf624890ad3c070"
I20260619 12:30:24.466938 7300 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d3a7477e1f1e4b52bdf624890ad3c070 in current term 2: Already voted for candidate 6520e328cbfb4ba8aefb2874c02d3dec in this term.
I20260619 12:30:24.466993 7123 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6520e328cbfb4ba8aefb2874c02d3dec in current term 2: Already voted for candidate d3a7477e1f1e4b52bdf624890ad3c070 in this term.
I20260619 12:30:24.467711 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:24.467526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:36164)
0619 12:30:24.467610 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:24.467654 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 12:30:24.467792 7284 leader_election.cc:334] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5163b08470e840a98fb193929dc7ad25 (127.0.98.62:43543): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
I20260619 12:30:24.467818 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:24.467595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:36178)
0619 12:30:24.467704 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:24.467749 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 12:30:24.467849 7284 leader_election.cc:302] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6520e328cbfb4ba8aefb2874c02d3dec; no voters: 5163b08470e840a98fb193929dc7ad25, d3a7477e1f1e4b52bdf624890ad3c070
W20260619 12:30:24.467907 7100 leader_election.cc:334] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5163b08470e840a98fb193929dc7ad25 (127.0.98.62:43543): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
I20260619 12:30:24.467962 7100 leader_election.cc:302] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3a7477e1f1e4b52bdf624890ad3c070; no voters: 5163b08470e840a98fb193929dc7ad25, 6520e328cbfb4ba8aefb2874c02d3dec
I20260619 12:30:24.467972 7822 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 12:30:24.468091 7823 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 12:30:24.952626 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:24.952120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.1:38815)
0619 12:30:24.952423 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:24.952506 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":99,"threads_started":1}
I20260619 12:30:24.954041 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:24.953700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.2:36721)
0619 12:30:24.953915 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:24.953969 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
I20260619 12:30:24.955009 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:24.954683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.3:46705)
0619 12:30:24.954890 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:24.954939 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
I20260619 12:30:25.953903 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:25.953390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.1:52373)
0619 12:30:25.953707 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:25.953789 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":80,"threads_started":1}
I20260619 12:30:25.955021 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:25.954658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.2:48693)
0619 12:30:25.954893 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:25.954945 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":88,"threads_started":1}
I20260619 12:30:25.955932 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:25.955593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.3:49357)
0619 12:30:25.955798 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:25.955851 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":57,"threads_started":1}
I20260619 12:30:25.969576 7838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:30:25.969707 7838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } }
I20260619 12:30:25.970163 7838 leader_election.cc:288] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5163b08470e840a98fb193929dc7ad25 (127.0.98.62:43543), d3a7477e1f1e4b52bdf624890ad3c070 (127.0.98.61:34171)
I20260619 12:30:25.970863 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:25.970409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:36384)
0619 12:30:25.970656 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:25.970721 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":70,"threads_started":1}
I20260619 12:30:25.970978 7123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" is_pre_election: true
I20260619 12:30:25.971139 7123 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6520e328cbfb4ba8aefb2874c02d3dec in term 2.
I20260619 12:30:25.971705 7284 leader_election.cc:302] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6520e328cbfb4ba8aefb2874c02d3dec, d3a7477e1f1e4b52bdf624890ad3c070; no voters:
I20260619 12:30:25.971877 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:25.971597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:36394)
0619 12:30:25.971712 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:25.971763 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 12:30:25.971896 7838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260619 12:30:25.971984 7284 leader_election.cc:334] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5163b08470e840a98fb193929dc7ad25 (127.0.98.62:43543): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
I20260619 12:30:25.972003 7838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:30:25.972097 7838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 2 FOLLOWER]: Advancing to term 3
I20260619 12:30:25.973272 7838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } }
I20260619 12:30:25.973599 7838 leader_election.cc:288] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [CANDIDATE]: Term 3 election: Requested vote from peers 5163b08470e840a98fb193929dc7ad25 (127.0.98.62:43543), d3a7477e1f1e4b52bdf624890ad3c070 (127.0.98.61:34171)
I20260619 12:30:25.974026 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:25.973767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:36404)
0619 12:30:25.973910 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:25.973965 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 12:30:25.974175 7123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d3a7477e1f1e4b52bdf624890ad3c070"
I20260619 12:30:25.974335 7123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 2 FOLLOWER]: Advancing to term 3
I20260619 12:30:25.974661 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:25.974489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:36414)
0619 12:30:25.974572 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:25.974618 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 12:30:25.974838 7284 leader_election.cc:334] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 5163b08470e840a98fb193929dc7ad25 (127.0.98.62:43543): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
I20260619 12:30:25.975500 7123 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6520e328cbfb4ba8aefb2874c02d3dec in term 3.
I20260619 12:30:25.975782 7284 leader_election.cc:302] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6520e328cbfb4ba8aefb2874c02d3dec, d3a7477e1f1e4b52bdf624890ad3c070; no voters: 5163b08470e840a98fb193929dc7ad25
I20260619 12:30:25.975943 7838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 3 FOLLOWER]: Leader election won for term 3
I20260619 12:30:25.976143 7838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [term 3 LEADER]: Becoming Leader. State: Replica: 6520e328cbfb4ba8aefb2874c02d3dec, State: Running, Role: LEADER
I20260619 12:30:25.976430 7838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } }
I20260619 12:30:25.977861 7841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6520e328cbfb4ba8aefb2874c02d3dec. Latest consensus state: current_term: 3 leader_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } } }
I20260619 12:30:25.977969 7841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:25.978245 7843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:30:25.980419 7843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:30:25.980790 7843 catalog_manager.cc:1259] Loaded cluster ID: 1e3227015b29436ab0201074af2dcaf3
I20260619 12:30:25.980840 7843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:30:25.981454 7843 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:30:25.981832 7843 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: Loaded TSK: 0
I20260619 12:30:25.982471 7843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:30:26.046474 7292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:30:26.049598 7292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:30:26.052528 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.052280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:36430)
0619 12:30:26.052415 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:26.052463 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 12:30:26.052539 7123 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [term 3 FOLLOWER]: Refusing update from remote peer 6520e328cbfb4ba8aefb2874c02d3dec: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 12:30:26.053025 7838 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 12:30:26.053352 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.053151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.0.1:36442)
0619 12:30:26.053243 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:26.053293 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 12:30:26.053491 7284 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec -> Peer 5163b08470e840a98fb193929dc7ad25 (127.0.98.62:43543): Couldn't send request to peer 5163b08470e840a98fb193929dc7ad25. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 12:30:26.054669 7845 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6520e328cbfb4ba8aefb2874c02d3dec. Latest consensus state: current_term: 3 leader_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } } }
I20260619 12:30:26.054801 7845 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:30:26.055835 7841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } } }
I20260619 12:30:26.055948 7841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:26.056210 7845 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5163b08470e840a98fb193929dc7ad25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43543 } } peers { permanent_uuid: "d3a7477e1f1e4b52bdf624890ad3c070" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34171 } attrs { promote: false } } peers { permanent_uuid: "6520e328cbfb4ba8aefb2874c02d3dec" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32971 } attrs { promote: false } } }
I20260619 12:30:26.056324 7845 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a7477e1f1e4b52bdf624890ad3c070 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:30:26.066144 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.062301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34403 (local address 127.0.0.1:37704)
0619 12:30:26.062688 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:26.062701 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:26.062780 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:26.063140 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:26.063143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:26.063159 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:26.063237 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:26.063241 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:26.064096 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:26.064102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:26.064899 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:26.064907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:26.065641 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:26.065645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:26.065731 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:26.065763 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 12:30:26.065833 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 12:30:26.065968 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":51,"thread_start_us":77,"threads_started":1}
I20260619 12:30:26.066270 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.061790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42353 (local address 127.0.0.1:42126)
0619 12:30:26.061958 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:26.061973 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:26.062062 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:26.062421 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:26.062427 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:26.062442 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:26.062577 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:26.062584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:26.063687 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:26.063690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:26.064922 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:26.064931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:26.065788 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:26.065794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:26.065926 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:26.065947 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 12:30:26.066031 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 12:30:26.066097 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":52}
I20260619 12:30:26.066746 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.061898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42126 (local address 127.0.98.1:42353)
0619 12:30:26.062149 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:26.062156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:26.062172 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:30:26.062228 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:26.062231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:26.062285 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:26.062686 (+ 401us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:26.062814 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:26.063585 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:26.065054 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:26.065699 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:26.065798 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:26.065863 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:26.066497 (+ 634us) server_negotiation.cc:299] Negotiation successful
0619 12:30:26.066582 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20260619 12:30:26.067083 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.061957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34003 (local address 127.0.0.1:45318)
0619 12:30:26.062255 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:26.062268 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:26.062356 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:26.063286 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:26.063289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:26.063305 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:26.063394 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:26.063400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:26.064444 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:26.064448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:26.065583 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:26.065593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:26.066706 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:26.066710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:26.066818 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:26.066843 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 12:30:26.066897 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 12:30:26.066960 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":104,"threads_started":1}
I20260619 12:30:26.067098 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.062624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37704 (local address 127.0.98.3:34403)
0619 12:30:26.062854 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:26.062858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:26.062870 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:30:26.062913 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:26.062916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:26.062958 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:26.063040 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:26.063353 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:26.063991 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:26.065017 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:26.065525 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:26.065651 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:26.065713 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:26.066899 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0619 12:30:26.066976 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":65,"threads_started":1}
I20260619 12:30:26.067621 7685 tablet_service.cc:1467] Processing CreateTablet for tablet ea1d7d30cc324eeea5f00bbf8cd6243a (DEFAULT_TABLE table=test-table [id=12ca3dd9a02d41e8990c790f8c83bf74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:26.067629 7411 tablet_service.cc:1467] Processing CreateTablet for tablet ea1d7d30cc324eeea5f00bbf8cd6243a (DEFAULT_TABLE table=test-table [id=12ca3dd9a02d41e8990c790f8c83bf74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:26.067901 7411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea1d7d30cc324eeea5f00bbf8cd6243a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:26.067901 7685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea1d7d30cc324eeea5f00bbf8cd6243a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:26.070694 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.062307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.0.98.2:34003)
0619 12:30:26.062986 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:26.062993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:26.063009 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:30:26.063060 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:26.063063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:26.063168 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:26.063277 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:26.063507 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:26.064320 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:26.065927 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:26.066573 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:26.070451 (+ 3878us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:26.070502 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:26.070534 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 12:30:26.070588 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":78,"threads_started":1}
I20260619 12:30:26.070842 7856 tablet_bootstrap.cc:492] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc: Bootstrap starting.
I20260619 12:30:26.071177 7548 tablet_service.cc:1467] Processing CreateTablet for tablet ea1d7d30cc324eeea5f00bbf8cd6243a (DEFAULT_TABLE table=test-table [id=12ca3dd9a02d41e8990c790f8c83bf74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:26.071439 7548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea1d7d30cc324eeea5f00bbf8cd6243a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:26.071790 7856 tablet_bootstrap.cc:654] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:26.072410 7857 tablet_bootstrap.cc:492] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2: Bootstrap starting.
I20260619 12:30:26.073019 7856 tablet_bootstrap.cc:492] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc: No bootstrap required, opened a new log
I20260619 12:30:26.073133 7856 ts_tablet_manager.cc:1397] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:30:26.073330 7857 tablet_bootstrap.cc:654] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:26.073591 7856 raft_consensus.cc:348] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } } peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } }
I20260619 12:30:26.073823 7856 raft_consensus.cc:374] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:26.073923 7856 raft_consensus.cc:729] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb3e17ffe404886b679448d181926dc, State: Initialized, Role: FOLLOWER
I20260619 12:30:26.074120 7856 consensus_queue.cc:260] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } } peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } }
I20260619 12:30:26.074278 7859 tablet_bootstrap.cc:492] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f: Bootstrap starting.
I20260619 12:30:26.074563 7856 ts_tablet_manager.cc:1428] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:30:26.074676 7857 tablet_bootstrap.cc:492] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2: No bootstrap required, opened a new log
I20260619 12:30:26.074759 7857 ts_tablet_manager.cc:1397] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:30:26.075119 7857 raft_consensus.cc:348] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } } peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } }
I20260619 12:30:26.075242 7857 raft_consensus.cc:374] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:26.075273 7857 raft_consensus.cc:729] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79139bcb3a2c449996505234878d5ed2, State: Initialized, Role: FOLLOWER
I20260619 12:30:26.075387 7857 consensus_queue.cc:260] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } } peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } }
I20260619 12:30:26.075608 7859 tablet_bootstrap.cc:654] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:26.075733 7857 ts_tablet_manager.cc:1428] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:30:26.076375 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.074838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.3:42231)
0619 12:30:26.076245 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:26.076305 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1330}
I20260619 12:30:26.077572 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.077302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.1:54945)
0619 12:30:26.077441 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:26.077497 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 12:30:26.078099 7477 heartbeater.cc:503] Master 127.0.98.60:32971 was elected leader, sending a full tablet report...
I20260619 12:30:26.078502 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.078239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.3:43151)
0619 12:30:26.078383 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:26.078436 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 12:30:26.078775 7859 tablet_bootstrap.cc:492] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f: No bootstrap required, opened a new log
I20260619 12:30:26.078892 7859 ts_tablet_manager.cc:1397] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 12:30:26.079316 7859 raft_consensus.cc:348] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } } peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } }
I20260619 12:30:26.079437 7859 raft_consensus.cc:374] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:26.079486 7859 raft_consensus.cc:729] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d437cfac39e64c909043a1c57af6668f, State: Initialized, Role: FOLLOWER
I20260619 12:30:26.079617 7859 consensus_queue.cc:260] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } } peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } }
I20260619 12:30:26.079945 7859 ts_tablet_manager.cc:1428] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:30:26.080744 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.080302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.2:43483)
0619 12:30:26.080599 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:26.080659 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222}
I20260619 12:30:26.081379 7614 heartbeater.cc:503] Master 127.0.98.60:32971 was elected leader, sending a full tablet report...
I20260619 12:30:26.081802 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.081425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.2:59397)
0619 12:30:26.081694 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:26.081744 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
I20260619 12:30:26.152484 7862 raft_consensus.cc:482] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:30:26.152720 7862 raft_consensus.cc:504] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } } peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } }
I20260619 12:30:26.153201 7862 leader_election.cc:288] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d437cfac39e64c909043a1c57af6668f (127.0.98.2:34003), ecb3e17ffe404886b679448d181926dc (127.0.98.3:34403)
I20260619 12:30:26.153779 7568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea1d7d30cc324eeea5f00bbf8cd6243a" candidate_uuid: "79139bcb3a2c449996505234878d5ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d437cfac39e64c909043a1c57af6668f" is_pre_election: true
I20260619 12:30:26.153779 7705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea1d7d30cc324eeea5f00bbf8cd6243a" candidate_uuid: "79139bcb3a2c449996505234878d5ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb3e17ffe404886b679448d181926dc" is_pre_election: true
I20260619 12:30:26.153946 7568 raft_consensus.cc:2454] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79139bcb3a2c449996505234878d5ed2 in term 0.
I20260619 12:30:26.153946 7705 raft_consensus.cc:2454] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79139bcb3a2c449996505234878d5ed2 in term 0.
I20260619 12:30:26.154325 7367 leader_election.cc:302] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 79139bcb3a2c449996505234878d5ed2, d437cfac39e64c909043a1c57af6668f, ecb3e17ffe404886b679448d181926dc; no voters:
I20260619 12:30:26.154491 7862 raft_consensus.cc:2781] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:30:26.154557 7862 raft_consensus.cc:482] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:30:26.154608 7862 raft_consensus.cc:3037] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:26.155678 7862 raft_consensus.cc:504] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } } peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } }
I20260619 12:30:26.156025 7862 leader_election.cc:288] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [CANDIDATE]: Term 1 election: Requested vote from peers d437cfac39e64c909043a1c57af6668f (127.0.98.2:34003), ecb3e17ffe404886b679448d181926dc (127.0.98.3:34403)
I20260619 12:30:26.156426 7705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea1d7d30cc324eeea5f00bbf8cd6243a" candidate_uuid: "79139bcb3a2c449996505234878d5ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb3e17ffe404886b679448d181926dc"
I20260619 12:30:26.156467 7568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea1d7d30cc324eeea5f00bbf8cd6243a" candidate_uuid: "79139bcb3a2c449996505234878d5ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d437cfac39e64c909043a1c57af6668f"
I20260619 12:30:26.156536 7705 raft_consensus.cc:3037] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:26.156620 7568 raft_consensus.cc:3037] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:26.157723 7705 raft_consensus.cc:2454] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79139bcb3a2c449996505234878d5ed2 in term 1.
I20260619 12:30:26.157886 7568 raft_consensus.cc:2454] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79139bcb3a2c449996505234878d5ed2 in term 1.
I20260619 12:30:26.158044 7367 leader_election.cc:302] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79139bcb3a2c449996505234878d5ed2, ecb3e17ffe404886b679448d181926dc; no voters:
I20260619 12:30:26.158234 7862 raft_consensus.cc:2781] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:26.158452 7862 raft_consensus.cc:686] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [term 1 LEADER]: Becoming Leader. State: Replica: 79139bcb3a2c449996505234878d5ed2, State: Running, Role: LEADER
I20260619 12:30:26.158607 7862 consensus_queue.cc:237] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } } peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } }
I20260619 12:30:26.159194 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.158890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.1:44619)
0619 12:30:26.159050 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:26.159117 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 12:30:26.160368 7292 catalog_manager.cc:5495] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79139bcb3a2c449996505234878d5ed2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "79139bcb3a2c449996505234878d5ed2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79139bcb3a2c449996505234878d5ed2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 } health_report { overall_health: UNKNOWN } } }
I20260619 12:30:26.190351 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.187300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42140 (local address 127.0.98.1:42353)
0619 12:30:26.187456 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:26.187462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:26.187478 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:30:26.187529 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:26.187531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:26.187573 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:26.187666 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:30:26.187922 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:26.188427 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:26.189304 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:26.189513 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:26.189635 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:26.189874 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:30:26.189952 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:30:26.190032 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:26.190148 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 12:30:26.190201 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:30:26.192588 7705 raft_consensus.cc:1261] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc [term 1 FOLLOWER]: Refusing update from remote peer 79139bcb3a2c449996505234878d5ed2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:26.192682 7568 raft_consensus.cc:1261] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f [term 1 FOLLOWER]: Refusing update from remote peer 79139bcb3a2c449996505234878d5ed2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:26.193168 7862 consensus_queue.cc:1035] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecb3e17ffe404886b679448d181926dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:26.193524 7865 consensus_queue.cc:1035] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d437cfac39e64c909043a1c57af6668f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:26.194161 7751 heartbeater.cc:503] Master 127.0.98.60:32971 was elected leader, sending a full tablet report...
I20260619 12:30:26.195415 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.195006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.3:48503)
0619 12:30:26.195257 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:26.195320 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260619 12:30:26.196781 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.196397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.2:45233)
0619 12:30:26.196658 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:26.196710 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260619 12:30:26.197212 7868 mvcc.cc:205] Tried to move back new op lower bound from 7298548638480412672 to 7298548638347182080. Current Snapshot: MvccSnapshot[applied={T|T < 7298548638480412672}]
I20260619 12:30:26.197917 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.197456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.1:47027)
0619 12:30:26.197800 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:26.197853 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281}
I20260619 12:30:26.199189 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.198769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.2:42801)
0619 12:30:26.199049 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:26.199104 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260619 12:30:26.200475 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.200031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.1:44573)
0619 12:30:26.200358 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:26.200411 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268}
I20260619 12:30:26.201269 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.200837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43543 (local address 127.0.98.3:36387)
0619 12:30:26.201130 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:26.201182 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260619 12:30:26.206238 7292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:26.206368 7292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:26.209769 7292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6520e328cbfb4ba8aefb2874c02d3dec: Sending DeleteTablet for 3 replicas of tablet ea1d7d30cc324eeea5f00bbf8cd6243a
I20260619 12:30:26.210343 7411 tablet_service.cc:1514] Processing DeleteTablet for tablet ea1d7d30cc324eeea5f00bbf8cd6243a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:26 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:42126
I20260619 12:30:26.210613 7548 tablet_service.cc:1514] Processing DeleteTablet for tablet ea1d7d30cc324eeea5f00bbf8cd6243a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:26 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:45318
I20260619 12:30:26.210613 7685 tablet_service.cc:1514] Processing DeleteTablet for tablet ea1d7d30cc324eeea5f00bbf8cd6243a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:26 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:37704
I20260619 12:30:26.210968 7877 tablet_replica.cc:330] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2: stopping tablet replica
I20260619 12:30:26.210989 7878 tablet_replica.cc:330] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f: stopping tablet replica
I20260619 12:30:26.211120 7877 raft_consensus.cc:2229] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:30:26.211356 7877 raft_consensus.cc:2258] T ea1d7d30cc324eeea5f00bbf8cd6243a P 79139bcb3a2c449996505234878d5ed2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:26.211416 7879 tablet_replica.cc:330] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc: stopping tablet replica
I20260619 12:30:26.211544 7879 raft_consensus.cc:2229] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:26.211681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 7258
I20260619 12:30:26.211685 7878 raft_consensus.cc:2229] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:26.211849 7878 raft_consensus.cc:2258] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:26.212049 7879 raft_consensus.cc:2258] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:26.213145 7878 ts_tablet_manager.cc:1905] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:26.213281 7879 ts_tablet_manager.cc:1905] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:26.215703 7878 ts_tablet_manager.cc:1918] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:30:26.215705 7879 ts_tablet_manager.cc:1918] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:30:26.215782 7879 log.cc:1196] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/ea1d7d30cc324eeea5f00bbf8cd6243a
I20260619 12:30:26.215827 7878 log.cc:1196] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/ea1d7d30cc324eeea5f00bbf8cd6243a
I20260619 12:30:26.216131 7879 ts_tablet_manager.cc:1939] T ea1d7d30cc324eeea5f00bbf8cd6243a P ecb3e17ffe404886b679448d181926dc: Deleting consensus metadata
I20260619 12:30:26.216146 7878 ts_tablet_manager.cc:1939] T ea1d7d30cc324eeea5f00bbf8cd6243a P d437cfac39e64c909043a1c57af6668f: Deleting consensus metadata
I20260619 12:30:26.216806 7282 catalog_manager.cc:4841] TS d437cfac39e64c909043a1c57af6668f (127.0.98.2:34003): tablet ea1d7d30cc324eeea5f00bbf8cd6243a (table test-table [id=12ca3dd9a02d41e8990c790f8c83bf74]) successfully deleted
I20260619 12:30:26.216892 7285 catalog_manager.cc:4841] TS ecb3e17ffe404886b679448d181926dc (127.0.98.3:34403): tablet ea1d7d30cc324eeea5f00bbf8cd6243a (table test-table [id=12ca3dd9a02d41e8990c790f8c83bf74]) successfully deleted
I20260619 12:30:26.220062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 7485
I20260619 12:30:26.220964 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:26.220242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42353 (local address 127.0.0.1:42154)
0619 12:30:26.220837 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:26.220888 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518}
W20260619 12:30:26.221195 7285 catalog_manager.cc:4584] TS 79139bcb3a2c449996505234878d5ed2 (127.0.98.1:42353): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea1d7d30cc324eeea5f00bbf8cd6243a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42353: connect: Connection refused (error 111)
I20260619 12:30:26.226794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 7621
I20260619 12:30:26.232872 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 6977
I20260619 12:30:26.238452 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 7080
2026-06-19T12:30:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7060 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:30:26 dist-test-slave-fg3h krb5kdc[7882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:30:26 dist-test-slave-fg3h krb5kdc[7882](info): set up 2 sockets
Jun 19 12:30:26 dist-test-slave-fg3h krb5kdc[7882](info): commencing operation
krb5kdc: starting...
W20260619 12:30:27.313784 392 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:30:27 dist-test-slave-fg3h krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:30:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:30:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 12:30:27.440485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41555
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:27.551383 7895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:27.551719 7895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:27.551797 7895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:27.555958 7895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:27.556056 7895 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 12:30:27.556088 7895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:27.556131 7895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:27.556166 7895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:27.556223 7895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:30:27.561798 7895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41555
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:27.562945 7895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:27.564401 7895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:27.577229 7900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:27.577441 7895 server_base.cc:1034] running on GCE node
W20260619 12:30:27.577266 7903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:27.577203 7901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:27.578895 7895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:27.580178 7895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:27.581378 7895 hybrid_clock.cc:648] HybridClock initialized: now 1781872227581344 us; error 59 us; skew 500 ppm
Jun 19 12:30:27 dist-test-slave-fg3h krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:27.584575 7895 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 12:30:27.585530 7895 webserver.cc:460] Webserver started at http://127.0.98.62:33989/ using document root <none> and password file <none>
I20260619 12:30:27.586061 7895 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:27.586133 7895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:27.586336 7895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:27.588331 7895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "a8164e8dab6e456985235b59053ff683"
format_stamp: "Formatted at 2026-06-19 12:30:27 on dist-test-slave-fg3h"
server_key: "d8f4880fee5bd9d52f18d5fdf9f72527"
server_key_iv: "9940e93928139e189a44d33d794c6f45"
server_key_version: "encryptionkey@0"
I20260619 12:30:27.588841 7895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "a8164e8dab6e456985235b59053ff683"
format_stamp: "Formatted at 2026-06-19 12:30:27 on dist-test-slave-fg3h"
server_key: "d8f4880fee5bd9d52f18d5fdf9f72527"
server_key_iv: "9940e93928139e189a44d33d794c6f45"
server_key_version: "encryptionkey@0"
I20260619 12:30:27.592506 7895 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 12:30:27.594862 7910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:27.596119 7895 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:30:27.596217 7895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "a8164e8dab6e456985235b59053ff683"
format_stamp: "Formatted at 2026-06-19 12:30:27 on dist-test-slave-fg3h"
server_key: "d8f4880fee5bd9d52f18d5fdf9f72527"
server_key_iv: "9940e93928139e189a44d33d794c6f45"
server_key_version: "encryptionkey@0"
I20260619 12:30:27.596374 7895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:27.610248 7895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:27.613023 7895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:27.613284 7895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:27.620859 7895 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41555
I20260619 12:30:27.620934 7972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41555 every 8 connection(s)
I20260619 12:30:27.621829 7895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:30:27.624603 7973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:27.627141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 7895
I20260619 12:30:27.627278 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:30:27.627586 392 external_mini_cluster.cc:1428] Setting key f2dea225c471f3ff0532ffd7d3dd0f0d
I20260619 12:30:27.631131 7973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683: Bootstrap starting.
I20260619 12:30:27.633728 7973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:27.634505 7973 log.cc:824] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:27.636502 7973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683: No bootstrap required, opened a new log
Jun 19 12:30:27 dist-test-slave-fg3h krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:30:27.640935 7973 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8164e8dab6e456985235b59053ff683" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41555 } }
I20260619 12:30:27.641232 7973 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:27.641294 7973 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8164e8dab6e456985235b59053ff683, State: Initialized, Role: FOLLOWER
I20260619 12:30:27.641788 7973 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8164e8dab6e456985235b59053ff683" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41555 } }
I20260619 12:30:27.641947 7973 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:30:27.642040 7973 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:30:27.642139 7973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:27.643177 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:27.629047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.0.98.62:41555)
0619 12:30:27.629486 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:27.629544 (+ 58us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:27.629579 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:30:27.630308 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:27.630327 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:27.630646 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:27.630999 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:27.631891 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:27.633410 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:27.634386 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:27.635040 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:27.635159 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:27.637845 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:27.637851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:27.637867 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:27.637896 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:27.639954 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:27.640399 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:27.640405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:27.640411 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:27.640559 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:27.640839 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:27.640842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:27.640843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:27.641380 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:27.641571 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:27.641895 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 12:30:27.642199 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":132,"threads_started":1}
I20260619 12:30:27.643257 7973 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8164e8dab6e456985235b59053ff683" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41555 } }
I20260619 12:30:27.643582 7973 leader_election.cc:302] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8164e8dab6e456985235b59053ff683; no voters:
I20260619 12:30:27.643895 7973 leader_election.cc:288] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:30:27.644011 7978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:27.644330 7978 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683 [term 1 LEADER]: Becoming Leader. State: Replica: a8164e8dab6e456985235b59053ff683, State: Running, Role: LEADER
I20260619 12:30:27.644734 7978 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8164e8dab6e456985235b59053ff683" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41555 } }
I20260619 12:30:27.645147 7973 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:27.646493 7980 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8164e8dab6e456985235b59053ff683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8164e8dab6e456985235b59053ff683" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41555 } } }
I20260619 12:30:27.646626 7980 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:27.646934 7979 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8164e8dab6e456985235b59053ff683. Latest consensus state: current_term: 1 leader_uuid: "a8164e8dab6e456985235b59053ff683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8164e8dab6e456985235b59053ff683" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41555 } } }
I20260619 12:30:27.647053 7979 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:27.647141 7988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:30:27.649887 7988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:30:27.656114 7988 catalog_manager.cc:1347] Generated new cluster ID: df6d7e8427574da8ab14d7305a4f627f
I20260619 12:30:27.656185 7988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:30:27.676321 7988 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:30:27.677381 7988 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:30:27.685930 7988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683: Generated new TSK 0
I20260619 12:30:27.686864 7988 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 12:30:27.721021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41555
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:27.832504 7999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:27.832756 7999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:27.832803 7999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:27.836957 7999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:27.837023 7999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:27.837062 7999 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:30:27.837150 7999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:30:27.842425 7999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:27.843425 7999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:27.844448 7999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:27.854629 8004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:27.854590 8007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:27.854584 8005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:27.854995 7999 server_base.cc:1034] running on GCE node
I20260619 12:30:27.855444 7999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:27.856134 7999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:27.857331 7999 hybrid_clock.cc:648] HybridClock initialized: now 1781872227857296 us; error 46 us; skew 500 ppm
Jun 19 12:30:27 dist-test-slave-fg3h krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:27.860168 7999 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 12:30:27.861200 7999 webserver.cc:460] Webserver started at http://127.0.98.1:38965/ using document root <none> and password file <none>
I20260619 12:30:27.861753 7999 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:27.861814 7999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:27.861981 7999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:27.864287 7999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "b066104557584d178b9353215bcae60d"
format_stamp: "Formatted at 2026-06-19 12:30:27 on dist-test-slave-fg3h"
server_key: "a162d8f91310bdcee827a4ab6ca6086a"
server_key_iv: "edb553c45564b93950c3491fb404ddba"
server_key_version: "encryptionkey@0"
I20260619 12:30:27.864856 7999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b066104557584d178b9353215bcae60d"
format_stamp: "Formatted at 2026-06-19 12:30:27 on dist-test-slave-fg3h"
server_key: "a162d8f91310bdcee827a4ab6ca6086a"
server_key_iv: "edb553c45564b93950c3491fb404ddba"
server_key_version: "encryptionkey@0"
I20260619 12:30:27.868849 7999 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:30:27.871642 8014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:27.872867 7999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:30:27.873001 7999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "b066104557584d178b9353215bcae60d"
format_stamp: "Formatted at 2026-06-19 12:30:27 on dist-test-slave-fg3h"
server_key: "a162d8f91310bdcee827a4ab6ca6086a"
server_key_iv: "edb553c45564b93950c3491fb404ddba"
server_key_version: "encryptionkey@0"
I20260619 12:30:27.873162 7999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:27.895735 7999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:27.898747 7999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:27.898950 7999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:27.900492 7999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:27.900542 7999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:27.900604 7999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:27.900637 7999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:30:27 dist-test-slave-fg3h krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:30:27.911777 7999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41211
I20260619 12:30:27.912636 7999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:30:27.913463 8129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41211 every 8 connection(s)
I20260619 12:30:27.914350 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:27.901330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47083 (local address 127.0.98.62:41555)
0619 12:30:27.901520 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:27.901527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:27.902359 (+ 832us) server_negotiation.cc:407] Connection header received
0619 12:30:27.903646 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:27.903652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:27.903716 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:27.903792 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:27.905640 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:27.906260 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:27.907508 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:27.907764 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:27.907878 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:27.910507 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:27.910510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:27.910513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:27.910541 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:27.912237 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:27.912767 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:27.912770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:27.912772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:27.912856 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:27.913283 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:27.913287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:27.913289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:27.913524 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:27.913614 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:27.914113 (+ 499us) server_negotiation.cc:299] Negotiation successful
0619 12:30:27.914215 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 12:30:27.915061 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:27.901696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41555 (local address 127.0.98.1:47083)
0619 12:30:27.902179 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:27.902237 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:27.903377 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:27.904242 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:27.904254 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:27.904705 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:27.905466 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:27.905487 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:27.906392 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:27.906395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:27.907377 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:27.907391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:27.907925 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:27.907929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:27.908100 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:27.908689 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:27.908730 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:27.910343 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:27.912348 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:27.912353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:27.912366 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:27.912652 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:27.912953 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:27.912956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:27.912958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:27.913173 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:27.913671 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:27.913677 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:27.913885 (+ 208us) client_negotiation.cc:769] Sending connection context
0619 12:30:27.914028 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 12:30:27.914256 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":110,"threads_started":1}
I20260619 12:30:27.916235 8130 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41555
I20260619 12:30:27.916380 8130 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:27.916862 8130 heartbeater.cc:511] Master 127.0.98.62:41555 requested a full tablet report, sending...
I20260619 12:30:27.917712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 7999
I20260619 12:30:27.917848 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:30:27.918172 392 external_mini_cluster.cc:1428] Setting key 8b48f2d3393a97e4c20d8e81468c2240
I20260619 12:30:27.918499 7927 ts_manager.cc:194] Registered new tserver with Master: b066104557584d178b9353215bcae60d (127.0.98.1:41211)
I20260619 12:30:27.919857 7927 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47083
I20260619 12:30:27.926968 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:27.919328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52775 (local address 127.0.98.62:41555)
0619 12:30:27.919428 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:27.919436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:27.919446 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:30:27.919492 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:27.919497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:27.919535 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:27.919613 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:27.920115 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:27.920924 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:27.922268 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:27.922486 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:27.922576 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:27.923770 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:27.923772 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:27.923775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:27.923794 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:27.925328 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:27.925803 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:27.925805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:27.925806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:27.925873 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:27.926222 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:27.926224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:27.926225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:27.926374 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:27.926446 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:27.926768 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 12:30:27.926856 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 12:30:27.927148 8136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:27.918529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41555 (local address 127.0.98.1:52775)
0619 12:30:27.918818 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:27.918835 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:27.918918 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:27.919651 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:27.919654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:27.919868 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:27.919996 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:27.920004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:27.921049 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:27.921051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:27.922077 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:27.922090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:27.922608 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:27.922611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:27.922695 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:27.923168 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:27.923182 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:27.923631 (+ 449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:27.925430 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:27.925435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:27.925440 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:27.925708 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:27.925963 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:27.925968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:27.925973 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:27.926080 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:27.926467 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:27.926472 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:27.926588 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 12:30:27.926718 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 12:30:27.926882 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 12:30:27.948499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41555
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:28.059228 8139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:28.059551 8139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:28.059628 8139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:28.063772 8139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:28.063870 8139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:28.063948 8139 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:30:28.064031 8139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:30:28.069303 8139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:28.070386 8139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:28.071720 8139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:28.081653 8144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:28.081598 8147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:28.081624 8145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:28.081959 8139 server_base.cc:1034] running on GCE node
I20260619 12:30:28.082443 8139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:28.083134 8139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:28.084329 8139 hybrid_clock.cc:648] HybridClock initialized: now 1781872228084313 us; error 31 us; skew 500 ppm
Jun 19 12:30:28 dist-test-slave-fg3h krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:28.087332 8139 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 12:30:28.088193 8139 webserver.cc:460] Webserver started at http://127.0.98.2:41793/ using document root <none> and password file <none>
I20260619 12:30:28.088790 8139 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:28.088889 8139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:28.089175 8139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:28.091239 8139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "74fb4bf3fe99446a9ea930db56591837"
format_stamp: "Formatted at 2026-06-19 12:30:28 on dist-test-slave-fg3h"
server_key: "cbedeea20c18a10dfb79a21cc21fe372"
server_key_iv: "b4cf6bc21f049839aca5a97dd835d851"
server_key_version: "encryptionkey@0"
I20260619 12:30:28.091822 8139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "74fb4bf3fe99446a9ea930db56591837"
format_stamp: "Formatted at 2026-06-19 12:30:28 on dist-test-slave-fg3h"
server_key: "cbedeea20c18a10dfb79a21cc21fe372"
server_key_iv: "b4cf6bc21f049839aca5a97dd835d851"
server_key_version: "encryptionkey@0"
I20260619 12:30:28.095767 8139 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:30:28.098507 8154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:28.100164 8139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 12:30:28.100322 8139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "74fb4bf3fe99446a9ea930db56591837"
format_stamp: "Formatted at 2026-06-19 12:30:28 on dist-test-slave-fg3h"
server_key: "cbedeea20c18a10dfb79a21cc21fe372"
server_key_iv: "b4cf6bc21f049839aca5a97dd835d851"
server_key_version: "encryptionkey@0"
I20260619 12:30:28.100451 8139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:28.114059 8139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:28.118520 8139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:28.118726 8139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:28.120268 8139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:28.120321 8139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:28.120361 8139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:28.120383 8139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:30:28 dist-test-slave-fg3h krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:30:28.131644 8139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37411
I20260619 12:30:28.131733 8269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37411 every 8 connection(s)
I20260619 12:30:28.132658 8139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:30:28.134719 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.121255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50231 (local address 127.0.98.62:41555)
0619 12:30:28.121388 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:28.121392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:28.122502 (+ 1110us) server_negotiation.cc:407] Connection header received
0619 12:30:28.123640 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:28.123644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:28.123702 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:28.123796 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:28.125332 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.126000 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.127279 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.127615 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.127732 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.130845 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:28.130849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:28.130852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:28.130886 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:28.132404 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:28.133079 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:28.133084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:28.133086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:28.133218 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:28.133611 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:28.133614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:28.133616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:28.133795 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:28.133909 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:28.134431 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 12:30:28.134548 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:30:28.135501 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 8139
I20260619 12:30:28.135520 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.121807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41555 (local address 127.0.98.2:50231)
0619 12:30:28.122340 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:28.122383 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:28.123393 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:28.123969 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:28.123979 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:28.124442 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:28.125184 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.125205 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.126117 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.126123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.127145 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.127155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.127782 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.127786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.127997 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.128650 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:28.128670 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:28.130680 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:28.132524 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:28.132529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:28.132542 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:28.132922 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:28.133331 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:28.133334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:28.133336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:28.133497 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:28.133921 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:28.133929 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:28.134156 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 12:30:28.134338 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 12:30:28.134656 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":153,"threads_started":1}
I20260619 12:30:28.135670 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:30:28.135963 392 external_mini_cluster.cc:1428] Setting key e1c7c48826328b27d1538836e835c958
I20260619 12:30:28.137426 8270 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41555
I20260619 12:30:28.137566 8270 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:28.138162 8270 heartbeater.cc:511] Master 127.0.98.62:41555 requested a full tablet report, sending...
I20260619 12:30:28.139250 7927 ts_manager.cc:194] Registered new tserver with Master: 74fb4bf3fe99446a9ea930db56591837 (127.0.98.2:37411)
I20260619 12:30:28.140012 7927 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50231
I20260619 12:30:28.147639 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.139572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59251 (local address 127.0.98.62:41555)
0619 12:30:28.139696 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:28.139700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:28.139871 (+ 171us) server_negotiation.cc:407] Connection header received
0619 12:30:28.139960 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:28.139964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:28.140013 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:28.140098 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:28.140602 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.141353 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.142504 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.142785 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.142882 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.144167 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:28.144169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:28.144171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:28.144191 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:28.145667 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:28.146271 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:28.146275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:28.146277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:28.146398 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:28.146730 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:28.146732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:28.146734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:28.146929 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:28.147081 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:28.147383 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 12:30:28.147487 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:30:28.147646 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.139509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41555 (local address 127.0.98.2:59251)
0619 12:30:28.139781 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:28.139795 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:28.139888 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:28.140135 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:28.140138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:28.140363 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:28.140482 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.140490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.141482 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.141485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.142368 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.142381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.142904 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.142911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.143044 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.143557 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:28.143577 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:28.144033 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:28.145776 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:28.145781 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:28.145785 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:28.146109 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:28.146526 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:28.146530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:28.146532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:28.146622 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:28.147133 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:28.147141 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:28.147213 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 12:30:28.147323 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 12:30:28.147463 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 12:30:28.166425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41555
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:28.276727 8279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:28.277024 8279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:28.277127 8279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:28.281534 8279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:28.281589 8279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:28.281629 8279 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 12:30:28.281693 8279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:30:28.287001 8279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:28.288060 8279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:28.289134 8279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:28.298647 8285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:28.298702 8284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:28.298838 8279 server_base.cc:1034] running on GCE node
W20260619 12:30:28.298702 8287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:28.299437 8279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:28.300122 8279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:28.301314 8279 hybrid_clock.cc:648] HybridClock initialized: now 1781872228301298 us; error 27 us; skew 500 ppm
Jun 19 12:30:28 dist-test-slave-fg3h krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:28.303993 8279 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 12:30:28.304852 8279 webserver.cc:460] Webserver started at http://127.0.98.3:45295/ using document root <none> and password file <none>
I20260619 12:30:28.305430 8279 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:28.305492 8279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:28.305656 8279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:28.307612 8279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "ff8bd311e16c4523b87c4a29bfdd58d8"
format_stamp: "Formatted at 2026-06-19 12:30:28 on dist-test-slave-fg3h"
server_key: "16a108c55384fa5199d7ae90672abf7c"
server_key_iv: "e69eb0166deabd480e416bae3ae4f57b"
server_key_version: "encryptionkey@0"
I20260619 12:30:28.308123 8279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ff8bd311e16c4523b87c4a29bfdd58d8"
format_stamp: "Formatted at 2026-06-19 12:30:28 on dist-test-slave-fg3h"
server_key: "16a108c55384fa5199d7ae90672abf7c"
server_key_iv: "e69eb0166deabd480e416bae3ae4f57b"
server_key_version: "encryptionkey@0"
I20260619 12:30:28.311997 8279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 12:30:28.314486 8294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:28.315794 8279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:30:28.315912 8279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "ff8bd311e16c4523b87c4a29bfdd58d8"
format_stamp: "Formatted at 2026-06-19 12:30:28 on dist-test-slave-fg3h"
server_key: "16a108c55384fa5199d7ae90672abf7c"
server_key_iv: "e69eb0166deabd480e416bae3ae4f57b"
server_key_version: "encryptionkey@0"
I20260619 12:30:28.316120 8279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:28.332859 8279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:28.335940 8279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:28.336225 8279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:28.337975 8279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:28.338048 8279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:28.338140 8279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:28.338197 8279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 12:30:28 dist-test-slave-fg3h krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:30:28.349228 8279 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42283
I20260619 12:30:28.349489 8409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42283 every 8 connection(s)
I20260619 12:30:28.350307 8279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:30:28.352526 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.338723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32947 (local address 127.0.98.62:41555)
0619 12:30:28.338876 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:28.338882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:28.339808 (+ 926us) server_negotiation.cc:407] Connection header received
0619 12:30:28.341108 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:28.341112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:28.341178 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:28.341247 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:28.343124 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.343671 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.345611 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.346010 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.346121 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.348786 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:28.348789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:28.348792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:28.348835 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:28.350305 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:28.350963 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:28.350967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:28.350970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:28.351063 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:28.351480 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:28.351482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:28.351483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:28.351674 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:28.351755 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:28.352215 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 12:30:28.352335 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:30:28.352612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 8279
I20260619 12:30:28.352741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:30:28.353042 392 external_mini_cluster.cc:1428] Setting key 3c8b22ef79aed07bb3fd84ba4d009556
I20260619 12:30:28.353608 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.339123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41555 (local address 127.0.98.3:32947)
0619 12:30:28.339615 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:28.339684 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:28.340817 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:28.341714 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:28.341725 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:28.342219 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:28.342939 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.342962 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.343805 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.343811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.345454 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.345467 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.346179 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.346183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.346368 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.347065 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:28.347084 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:28.348617 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:28.350447 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:28.350452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:28.350465 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:28.350835 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:28.351168 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:28.351171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:28.351173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:28.351391 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:28.351765 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:28.351770 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:28.351980 (+ 210us) client_negotiation.cc:769] Sending connection context
0619 12:30:28.352132 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 12:30:28.352564 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":115,"threads_started":1}
I20260619 12:30:28.354873 8410 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41555
I20260619 12:30:28.355041 8410 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:28.355587 8410 heartbeater.cc:511] Master 127.0.98.62:41555 requested a full tablet report, sending...
I20260619 12:30:28.356571 7927 ts_manager.cc:194] Registered new tserver with Master: ff8bd311e16c4523b87c4a29bfdd58d8 (127.0.98.3:42283)
I20260619 12:30:28.357298 7927 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32947
I20260619 12:30:28.364149 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.356987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41555 (local address 127.0.98.3:46911)
0619 12:30:28.357289 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:28.357304 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:28.357412 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:28.357647 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:28.357650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:28.357866 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:28.357980 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.357987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.358722 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.358725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.359458 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.359473 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.359890 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.359892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.359957 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.360539 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:28.360560 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:28.360885 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:28.362678 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:28.362680 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:28.362682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:28.362926 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:28.363199 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:28.363202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:28.363204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:28.363339 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:28.363696 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:28.363698 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:28.363778 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 12:30:28.363864 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 12:30:28.363986 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":146,"threads_started":1}
I20260619 12:30:28.364166 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.357050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46911 (local address 127.0.98.62:41555)
0619 12:30:28.357196 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:28.357201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:28.357403 (+ 202us) server_negotiation.cc:407] Connection header received
0619 12:30:28.357487 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:28.357490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:28.357538 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:28.357630 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:28.358098 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.358576 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.359581 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.359784 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.359875 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.361004 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:28.361007 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:28.361009 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:28.361028 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:28.362569 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:28.363017 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:28.363020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:28.363021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:28.363092 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:28.363436 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:28.363438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:28.363439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:28.363602 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:28.363701 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:28.363951 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 12:30:28.364045 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 12:30:28.367692 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 12:30:28 dist-test-slave-fg3h krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872228, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 12:30:28 dist-test-slave-fg3h krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:28 dist-test-slave-fg3h krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 12:30:28.391209 7976 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:28.384963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45970 (local address 127.0.98.62:41555)
0619 12:30:28.385126 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:28.385131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:28.385151 (+ 20us) server_negotiation.cc:407] Connection header received
0619 12:30:28.385191 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:28.385193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:28.385231 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:28.385310 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:28.386055 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.386596 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.387564 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.387779 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.387858 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.391088 (+ 3230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45970: BlockingRecv error: recv got EOF from 127.0.0.1:45970 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 12:30:28 dist-test-slave-fg3h krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872228, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:30:28.400032 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.392608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45984 (local address 127.0.98.62:41555)
0619 12:30:28.392716 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:28.392720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:28.392817 (+ 97us) server_negotiation.cc:407] Connection header received
0619 12:30:28.392907 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:28.392908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:28.392942 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:28.393012 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:28.393603 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.394099 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.394840 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.395039 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.395118 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.397396 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:28.397399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:28.397401 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:28.397422 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:28.398626 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:28.398993 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:28.398996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:28.398997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:28.399070 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:28.399313 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:28.399315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:28.399316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:28.399471 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:28.399570 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:28.399835 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 12:30:28.399922 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 12:30:28.402808 7927 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:30:28.405963 7927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:30:28.421413 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.414611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42632 (local address 127.0.98.3:42283)
0619 12:30:28.415678 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:28.415692 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:28.415714 (+ 22us) server_negotiation.cc:407] Connection header received
0619 12:30:28.415803 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:28.415810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:28.416025 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:28.416239 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:28.416667 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.417901 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.419501 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.420375 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.420508 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.420678 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:28.421126 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 12:30:28.421234 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":965,"thread_start_us":128,"threads_started":1}
I20260619 12:30:28.421439 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.414870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41211 (local address 127.0.0.1:40578)
0619 12:30:28.416466 (+ 1596us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:28.416480 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:28.416563 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:28.416923 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:28.416926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:28.416942 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:28.417019 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.417025 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.418230 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.418235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.419517 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.419523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.420648 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.420652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.420808 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.420919 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 12:30:28.421062 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 12:30:28.421255 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1541,"mutex_wait_us":108,"thread_start_us":88,"threads_started":1}
I20260619 12:30:28.421450 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.414463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42283 (local address 127.0.0.1:42632)
0619 12:30:28.415204 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:28.415273 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:28.415405 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:28.416226 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:28.416233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:28.416260 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:28.416437 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.416444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.418014 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.418017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.419371 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.419380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.420648 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.420652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.420810 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.420957 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 12:30:28.421039 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 12:30:28.421248 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":225,"spinlock_wait_cycles":6016,"thread_start_us":66,"threads_started":1}
I20260619 12:30:28.422801 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.414601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37411 (local address 127.0.0.1:40798)
0619 12:30:28.415203 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:28.415263 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:28.415405 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:28.416420 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:28.416423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:28.416440 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:28.416549 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.416555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.418045 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.418048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.419502 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.419512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.422360 (+ 2848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.422364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.422472 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.422495 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:30:28.422578 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 12:30:28.422647 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":87,"spinlock_wait_cycles":3712,"thread_start_us":100,"threads_started":1}
I20260619 12:30:28.422909 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.415273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40578 (local address 127.0.98.1:41211)
0619 12:30:28.415733 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:28.415746 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:28.416583 (+ 837us) server_negotiation.cc:407] Connection header received
0619 12:30:28.416662 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:28.416672 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:28.416800 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:28.416944 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:28.417190 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.418115 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.419667 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.420428 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.420525 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.420665 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:28.422606 (+ 1941us) server_negotiation.cc:299] Negotiation successful
0619 12:30:28.422723 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":92,"threads_started":1}
I20260619 12:30:28.423209 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.415126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40798 (local address 127.0.98.2:37411)
0619 12:30:28.415866 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:28.415885 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:28.415902 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:30:28.415959 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:28.415963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:28.416191 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:28.416425 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:28.416677 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.417917 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.420772 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.422233 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.422351 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.422510 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:28.422979 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 12:30:28.423076 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":107,"threads_started":1}
I20260619 12:30:28.425120 8344 tablet_service.cc:1467] Processing CreateTablet for tablet ea6598ef4291418480337823a1e01f9b (DEFAULT_TABLE table=test-table [id=cb71c10e6bb842a9b026ee100a0293b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:28.425563 8064 tablet_service.cc:1467] Processing CreateTablet for tablet ea6598ef4291418480337823a1e01f9b (DEFAULT_TABLE table=test-table [id=cb71c10e6bb842a9b026ee100a0293b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:28.425664 8204 tablet_service.cc:1467] Processing CreateTablet for tablet ea6598ef4291418480337823a1e01f9b (DEFAULT_TABLE table=test-table [id=cb71c10e6bb842a9b026ee100a0293b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:28.426339 8344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea6598ef4291418480337823a1e01f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:28.426522 8064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea6598ef4291418480337823a1e01f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:28.426671 8204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea6598ef4291418480337823a1e01f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:28.433377 8438 tablet_bootstrap.cc:492] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837: Bootstrap starting.
I20260619 12:30:28.434310 8439 tablet_bootstrap.cc:492] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d: Bootstrap starting.
I20260619 12:30:28.434310 8440 tablet_bootstrap.cc:492] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8: Bootstrap starting.
I20260619 12:30:28.435607 8438 tablet_bootstrap.cc:654] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:28.436342 8438 log.cc:824] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:28.436359 8439 tablet_bootstrap.cc:654] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:28.436359 8440 tablet_bootstrap.cc:654] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:28.437254 8440 log.cc:824] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:28.437609 8439 log.cc:824] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:28.438621 8438 tablet_bootstrap.cc:492] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837: No bootstrap required, opened a new log
I20260619 12:30:28.438823 8438 ts_tablet_manager.cc:1397] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 12:30:28.439215 8440 tablet_bootstrap.cc:492] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8: No bootstrap required, opened a new log
I20260619 12:30:28.439424 8440 ts_tablet_manager.cc:1397] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260619 12:30:28.439463 8439 tablet_bootstrap.cc:492] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d: No bootstrap required, opened a new log
I20260619 12:30:28.439635 8439 ts_tablet_manager.cc:1397] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:30:28.442293 8440 raft_consensus.cc:348] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } }
I20260619 12:30:28.442528 8440 raft_consensus.cc:374] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:28.442579 8440 raft_consensus.cc:729] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8bd311e16c4523b87c4a29bfdd58d8, State: Initialized, Role: FOLLOWER
I20260619 12:30:28.442767 8438 raft_consensus.cc:348] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } }
I20260619 12:30:28.443014 8438 raft_consensus.cc:374] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:28.443077 8438 raft_consensus.cc:729] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fb4bf3fe99446a9ea930db56591837, State: Initialized, Role: FOLLOWER
I20260619 12:30:28.443027 8440 consensus_queue.cc:260] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } }
I20260619 12:30:28.443480 8438 consensus_queue.cc:260] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } }
I20260619 12:30:28.443652 8410 heartbeater.cc:503] Master 127.0.98.62:41555 was elected leader, sending a full tablet report...
I20260619 12:30:28.444020 8440 ts_tablet_manager.cc:1428] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260619 12:30:28.444419 8438 ts_tablet_manager.cc:1428] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 12:30:28.443354 8439 raft_consensus.cc:348] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } }
I20260619 12:30:28.444485 8439 raft_consensus.cc:374] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:28.444540 8439 raft_consensus.cc:729] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b066104557584d178b9353215bcae60d, State: Initialized, Role: FOLLOWER
I20260619 12:30:28.444736 8270 heartbeater.cc:503] Master 127.0.98.62:41555 was elected leader, sending a full tablet report...
I20260619 12:30:28.444931 8439 consensus_queue.cc:260] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } }
I20260619 12:30:28.445930 8439 ts_tablet_manager.cc:1428] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 12:30:28.445952 8130 heartbeater.cc:503] Master 127.0.98.62:41555 was elected leader, sending a full tablet report...
W20260619 12:30:28.601334 8411 tablet.cc:2369] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:30:28.633880 8271 tablet.cc:2369] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:30:28.664031 8131 tablet.cc:2369] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:30:28.756173 8446 raft_consensus.cc:482] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:30:28.756410 8446 raft_consensus.cc:504] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } }
I20260619 12:30:28.757565 8446 leader_election.cc:288] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff8bd311e16c4523b87c4a29bfdd58d8 (127.0.98.3:42283), 74fb4bf3fe99446a9ea930db56591837 (127.0.98.2:37411)
I20260619 12:30:28.761672 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.758110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57287 (local address 127.0.98.2:37411)
0619 12:30:28.758246 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:28.758250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:28.758462 (+ 212us) server_negotiation.cc:407] Connection header received
0619 12:30:28.758613 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:28.758617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:28.758668 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:28.758754 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:28.759050 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.759548 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.760676 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.761078 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.761217 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.761287 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:28.761471 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 12:30:28.761538 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 12:30:28.761718 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.757882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37411 (local address 127.0.98.1:57287)
0619 12:30:28.758345 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:28.758357 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:28.758504 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:28.758765 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:28.758768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:28.758782 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:28.758930 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.758935 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.759662 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.759667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.760543 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.760553 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.761223 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.761226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.761325 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.761347 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:30:28.761457 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 12:30:28.761528 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":126,"thread_start_us":92,"threads_started":1}
I20260619 12:30:28.761699 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.758036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37881 (local address 127.0.98.3:42283)
0619 12:30:28.758175 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:28.758181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:28.758423 (+ 242us) server_negotiation.cc:407] Connection header received
0619 12:30:28.758599 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:28.758603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:28.758649 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:28.758739 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:28.759032 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.759552 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.760687 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.761131 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.761252 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.761329 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:28.761507 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 12:30:28.761574 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:30:28.761749 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.757882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42283 (local address 127.0.98.1:37881)
0619 12:30:28.758309 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:28.758324 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:28.758467 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:28.758749 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:28.758751 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:28.758766 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:28.758915 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.758920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.759668 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.759674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.760558 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.760567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.761253 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.761257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.761377 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.761405 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 12:30:28.761499 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 12:30:28.761569 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":53,"spinlock_wait_cycles":9600,"thread_start_us":117,"threads_started":1}
I20260619 12:30:28.762388 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6598ef4291418480337823a1e01f9b" candidate_uuid: "b066104557584d178b9353215bcae60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fb4bf3fe99446a9ea930db56591837" is_pre_election: true
I20260619 12:30:28.762396 8364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6598ef4291418480337823a1e01f9b" candidate_uuid: "b066104557584d178b9353215bcae60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" is_pre_election: true
I20260619 12:30:28.762696 8224 raft_consensus.cc:2454] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b066104557584d178b9353215bcae60d in term 0.
I20260619 12:30:28.762702 8364 raft_consensus.cc:2454] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b066104557584d178b9353215bcae60d in term 0.
I20260619 12:30:28.763206 8015 leader_election.cc:302] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b066104557584d178b9353215bcae60d, ff8bd311e16c4523b87c4a29bfdd58d8; no voters:
I20260619 12:30:28.763513 8446 raft_consensus.cc:2781] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:30:28.763607 8446 raft_consensus.cc:482] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:30:28.763651 8446 raft_consensus.cc:3037] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:28.764729 8446 raft_consensus.cc:504] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } }
I20260619 12:30:28.765076 8446 leader_election.cc:288] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [CANDIDATE]: Term 1 election: Requested vote from peers ff8bd311e16c4523b87c4a29bfdd58d8 (127.0.98.3:42283), 74fb4bf3fe99446a9ea930db56591837 (127.0.98.2:37411)
I20260619 12:30:28.765515 8364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6598ef4291418480337823a1e01f9b" candidate_uuid: "b066104557584d178b9353215bcae60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8"
I20260619 12:30:28.765547 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6598ef4291418480337823a1e01f9b" candidate_uuid: "b066104557584d178b9353215bcae60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fb4bf3fe99446a9ea930db56591837"
I20260619 12:30:28.765623 8364 raft_consensus.cc:3037] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:28.765694 8224 raft_consensus.cc:3037] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:28.766752 8224 raft_consensus.cc:2454] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b066104557584d178b9353215bcae60d in term 1.
I20260619 12:30:28.767073 8017 leader_election.cc:302] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74fb4bf3fe99446a9ea930db56591837, b066104557584d178b9353215bcae60d; no voters:
I20260619 12:30:28.767076 8364 raft_consensus.cc:2454] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b066104557584d178b9353215bcae60d in term 1.
I20260619 12:30:28.767379 8446 raft_consensus.cc:2781] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:28.767649 8446 raft_consensus.cc:686] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [term 1 LEADER]: Becoming Leader. State: Replica: b066104557584d178b9353215bcae60d, State: Running, Role: LEADER
I20260619 12:30:28.768050 8446 consensus_queue.cc:237] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } }
I20260619 12:30:28.771436 7927 catalog_manager.cc:5495] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d reported cstate change: term changed from 0 to 1, leader changed from <none> to b066104557584d178b9353215bcae60d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b066104557584d178b9353215bcae60d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } health_report { overall_health: HEALTHY } } }
I20260619 12:30:28.784082 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.780207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40580 (local address 127.0.98.1:41211)
0619 12:30:28.780340 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:28.780344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:28.780358 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:30:28.780554 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:28.780557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:28.780618 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:28.780718 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:30:28.780949 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.781591 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.782485 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.782740 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.782848 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.783147 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:30:28.783624 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:30:28.783721 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:28.783824 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 12:30:28.783879 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:30:28.790424 8224 raft_consensus.cc:1261] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837 [term 1 FOLLOWER]: Refusing update from remote peer b066104557584d178b9353215bcae60d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:28.790424 8364 raft_consensus.cc:1261] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8 [term 1 FOLLOWER]: Refusing update from remote peer b066104557584d178b9353215bcae60d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:28.791235 8449 consensus_queue.cc:1035] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:28.791503 8446 consensus_queue.cc:1035] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:28.800524 8454 mvcc.cc:205] Tried to move back new op lower bound from 7298548649117642752 to 7298548649036816384. Current Snapshot: MvccSnapshot[applied={T|T < 7298548649117642752}]
I20260619 12:30:28.810742 7927 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:28.810921 7927 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:28.813212 7927 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683: Sending DeleteTablet for 3 replicas of tablet ea6598ef4291418480337823a1e01f9b
I20260619 12:30:28.813997 8204 tablet_service.cc:1514] Processing DeleteTablet for tablet ea6598ef4291418480337823a1e01f9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40798
I20260619 12:30:28.813997 8344 tablet_service.cc:1514] Processing DeleteTablet for tablet ea6598ef4291418480337823a1e01f9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42632
I20260619 12:30:28.814386 8064 tablet_service.cc:1514] Processing DeleteTablet for tablet ea6598ef4291418480337823a1e01f9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40578
I20260619 12:30:28.814486 8462 tablet_replica.cc:330] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837: stopping tablet replica
I20260619 12:30:28.814811 8462 raft_consensus.cc:2229] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:28.814956 8464 tablet_replica.cc:330] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d: stopping tablet replica
I20260619 12:30:28.815128 8462 raft_consensus.cc:2258] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:28.815315 8464 raft_consensus.cc:2229] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:30:28.815467 7927 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 12:30:28.815888 8464 raft_consensus.cc:2258] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 12:30:28.815920 7927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:30:28.816776 8462 ts_tablet_manager.cc:1905] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:28.817498 8463 tablet_replica.cc:330] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8: stopping tablet replica
I20260619 12:30:28.818084 8464 ts_tablet_manager.cc:1905] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:28.819515 8463 raft_consensus.cc:2229] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:28.820878 8064 tablet_service.cc:1467] Processing CreateTablet for tablet c9830afa7b2f45eb9ea71c9c4b27affa (DEFAULT_TABLE table=test-table [id=035d01fc54a24bea8af6a176912a8683]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:28.821185 8064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9830afa7b2f45eb9ea71c9c4b27affa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:28.821166 8464 ts_tablet_manager.cc:1918] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:30:28.821458 8464 log.cc:1196] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/wal/wals/ea6598ef4291418480337823a1e01f9b
I20260619 12:30:28.821484 8344 tablet_service.cc:1467] Processing CreateTablet for tablet c9830afa7b2f45eb9ea71c9c4b27affa (DEFAULT_TABLE table=test-table [id=035d01fc54a24bea8af6a176912a8683]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:28.821743 8344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9830afa7b2f45eb9ea71c9c4b27affa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:28.821993 8464 ts_tablet_manager.cc:1939] T ea6598ef4291418480337823a1e01f9b P b066104557584d178b9353215bcae60d: Deleting consensus metadata
I20260619 12:30:28.823266 7914 catalog_manager.cc:4841] TS b066104557584d178b9353215bcae60d (127.0.98.1:41211): tablet ea6598ef4291418480337823a1e01f9b (table test-table [id=cb71c10e6bb842a9b026ee100a0293b0]) successfully deleted
I20260619 12:30:28.823400 8439 tablet_bootstrap.cc:492] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d: Bootstrap starting.
I20260619 12:30:28.823997 8204 tablet_service.cc:1467] Processing CreateTablet for tablet c9830afa7b2f45eb9ea71c9c4b27affa (DEFAULT_TABLE table=test-table [id=035d01fc54a24bea8af6a176912a8683]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:28.824273 8204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9830afa7b2f45eb9ea71c9c4b27affa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:28.824290 8440 tablet_bootstrap.cc:492] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8: Bootstrap starting.
I20260619 12:30:28.824383 8439 tablet_bootstrap.cc:654] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:28.825477 8463 raft_consensus.cc:2258] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:28.825793 8440 tablet_bootstrap.cc:654] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:28.826911 8438 tablet_bootstrap.cc:492] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837: Bootstrap starting.
I20260619 12:30:28.827857 8438 tablet_bootstrap.cc:654] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:28.827968 8463 ts_tablet_manager.cc:1905] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:28.828379 8462 ts_tablet_manager.cc:1918] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:30:28.828508 8462 log.cc:1196] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/ea6598ef4291418480337823a1e01f9b
I20260619 12:30:28.829404 8462 ts_tablet_manager.cc:1939] T ea6598ef4291418480337823a1e01f9b P 74fb4bf3fe99446a9ea930db56591837: Deleting consensus metadata
I20260619 12:30:28.829481 8440 tablet_bootstrap.cc:492] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8: No bootstrap required, opened a new log
I20260619 12:30:28.829568 8440 ts_tablet_manager.cc:1397] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 12:30:28.830015 8440 raft_consensus.cc:348] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:28.830134 8440 raft_consensus.cc:374] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:28.830195 8440 raft_consensus.cc:729] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8bd311e16c4523b87c4a29bfdd58d8, State: Initialized, Role: FOLLOWER
I20260619 12:30:28.830345 8440 consensus_queue.cc:260] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:28.830598 8440 ts_tablet_manager.cc:1428] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 12:30:28.830941 8463 ts_tablet_manager.cc:1918] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:30:28.831036 8463 log.cc:1196] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/ea6598ef4291418480337823a1e01f9b
I20260619 12:30:28.831341 8463 ts_tablet_manager.cc:1939] T ea6598ef4291418480337823a1e01f9b P ff8bd311e16c4523b87c4a29bfdd58d8: Deleting consensus metadata
I20260619 12:30:28.831794 7913 catalog_manager.cc:4841] TS 74fb4bf3fe99446a9ea930db56591837 (127.0.98.2:37411): tablet ea6598ef4291418480337823a1e01f9b (table test-table [id=cb71c10e6bb842a9b026ee100a0293b0]) successfully deleted
I20260619 12:30:28.832957 8439 tablet_bootstrap.cc:492] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d: No bootstrap required, opened a new log
I20260619 12:30:28.833055 8439 ts_tablet_manager.cc:1397] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260619 12:30:28.833307 7911 catalog_manager.cc:4841] TS ff8bd311e16c4523b87c4a29bfdd58d8 (127.0.98.3:42283): tablet ea6598ef4291418480337823a1e01f9b (table test-table [id=cb71c10e6bb842a9b026ee100a0293b0]) successfully deleted
I20260619 12:30:28.833644 8438 tablet_bootstrap.cc:492] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837: No bootstrap required, opened a new log
I20260619 12:30:28.833581 8439 raft_consensus.cc:348] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:28.833698 8439 raft_consensus.cc:374] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:28.833746 8439 raft_consensus.cc:729] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b066104557584d178b9353215bcae60d, State: Initialized, Role: FOLLOWER
I20260619 12:30:28.833755 8438 ts_tablet_manager.cc:1397] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260619 12:30:28.833887 8439 consensus_queue.cc:260] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:28.834144 8439 ts_tablet_manager.cc:1428] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:30:28.834200 8438 raft_consensus.cc:348] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:28.834326 8438 raft_consensus.cc:374] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:28.834372 8438 raft_consensus.cc:729] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fb4bf3fe99446a9ea930db56591837, State: Initialized, Role: FOLLOWER
I20260619 12:30:28.834501 8438 consensus_queue.cc:260] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:28.834797 8438 ts_tablet_manager.cc:1428] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 12:30:28.874948 8445 raft_consensus.cc:482] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:30:28.875116 8445 raft_consensus.cc:504] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:28.876489 8445 leader_election.cc:288] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b066104557584d178b9353215bcae60d (127.0.98.1:41211), ff8bd311e16c4523b87c4a29bfdd58d8 (127.0.98.3:42283)
I20260619 12:30:28.881520 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.876917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36599 (local address 127.0.98.3:42283)
0619 12:30:28.877046 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:28.877051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:28.877215 (+ 164us) server_negotiation.cc:407] Connection header received
0619 12:30:28.877492 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:28.877497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:28.877574 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:28.877648 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:28.878097 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.878956 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.880307 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.880777 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.880904 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.880971 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:28.881296 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 12:30:28.881382 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:30:28.881651 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.876748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41211 (local address 127.0.98.2:54477)
0619 12:30:28.877274 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:28.877289 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:28.877405 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:28.877793 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:28.877797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:28.877819 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:28.877961 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.877966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.878696 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.878702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.880029 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.880038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.881022 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.881025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.881155 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.881174 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 12:30:28.881414 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 12:30:28.881487 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":48,"thread_start_us":90,"threads_started":1}
I20260619 12:30:28.881881 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.876740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42283 (local address 127.0.98.2:36599)
0619 12:30:28.877121 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:28.877137 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:28.877372 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:28.877679 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:28.877682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:28.877700 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:28.877957 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.877965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.879065 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.879068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.880180 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.880186 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.880912 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.880915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.881016 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.881038 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:30:28.881658 (+ 620us) client_negotiation.cc:240] Negotiation successful
0619 12:30:28.881732 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":9,"thread_start_us":73,"threads_started":1}
I20260619 12:30:28.882341 8364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9830afa7b2f45eb9ea71c9c4b27affa" candidate_uuid: "74fb4bf3fe99446a9ea930db56591837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" is_pre_election: true
I20260619 12:30:28.882476 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.877145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54477 (local address 127.0.98.1:41211)
0619 12:30:28.877272 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:28.877277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:28.877379 (+ 102us) server_negotiation.cc:407] Connection header received
0619 12:30:28.877569 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:28.877572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:28.877626 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:28.877700 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:28.878074 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.878588 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.880178 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.880855 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.880971 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.881038 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:28.882268 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0619 12:30:28.882350 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 12:30:28.882534 8364 raft_consensus.cc:2454] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74fb4bf3fe99446a9ea930db56591837 in term 0.
I20260619 12:30:28.882922 8155 leader_election.cc:302] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74fb4bf3fe99446a9ea930db56591837, ff8bd311e16c4523b87c4a29bfdd58d8; no voters:
I20260619 12:30:28.882957 8084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9830afa7b2f45eb9ea71c9c4b27affa" candidate_uuid: "74fb4bf3fe99446a9ea930db56591837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b066104557584d178b9353215bcae60d" is_pre_election: true
I20260619 12:30:28.883196 8084 raft_consensus.cc:2454] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74fb4bf3fe99446a9ea930db56591837 in term 0.
I20260619 12:30:28.883456 8445 raft_consensus.cc:2781] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:30:28.883528 8445 raft_consensus.cc:482] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:30:28.883582 8445 raft_consensus.cc:3037] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:28.884896 8445 raft_consensus.cc:504] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:28.885345 8445 leader_election.cc:288] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [CANDIDATE]: Term 1 election: Requested vote from peers b066104557584d178b9353215bcae60d (127.0.98.1:41211), ff8bd311e16c4523b87c4a29bfdd58d8 (127.0.98.3:42283)
I20260619 12:30:28.885741 8084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9830afa7b2f45eb9ea71c9c4b27affa" candidate_uuid: "74fb4bf3fe99446a9ea930db56591837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b066104557584d178b9353215bcae60d"
I20260619 12:30:28.885767 8364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9830afa7b2f45eb9ea71c9c4b27affa" candidate_uuid: "74fb4bf3fe99446a9ea930db56591837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8"
I20260619 12:30:28.885872 8364 raft_consensus.cc:3037] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:28.885872 8084 raft_consensus.cc:3037] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:28.887054 8084 raft_consensus.cc:2454] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74fb4bf3fe99446a9ea930db56591837 in term 1.
I20260619 12:30:28.887054 8364 raft_consensus.cc:2454] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74fb4bf3fe99446a9ea930db56591837 in term 1.
I20260619 12:30:28.887349 8158 leader_election.cc:302] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74fb4bf3fe99446a9ea930db56591837, b066104557584d178b9353215bcae60d; no voters:
I20260619 12:30:28.887537 8445 raft_consensus.cc:2781] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:28.887766 8445 raft_consensus.cc:686] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [term 1 LEADER]: Becoming Leader. State: Replica: 74fb4bf3fe99446a9ea930db56591837, State: Running, Role: LEADER
I20260619 12:30:28.888051 8445 consensus_queue.cc:237] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:28.890704 7926 catalog_manager.cc:5495] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74fb4bf3fe99446a9ea930db56591837 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "74fb4bf3fe99446a9ea930db56591837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } health_report { overall_health: UNKNOWN } } }
Jun 19 12:30:28 dist-test-slave-fg3h krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:30:28.927181 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.914978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45988 (local address 127.0.98.62:41555)
0619 12:30:28.915198 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:28.915210 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:28.915303 (+ 93us) server_negotiation.cc:407] Connection header received
0619 12:30:28.915456 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:28.915461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:28.915504 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:28.915599 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:28.916250 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.916886 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.917731 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.917950 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.918056 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.923817 (+ 5761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:28.923821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:28.923823 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:28.923845 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:28.925443 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:28.925923 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:28.925929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:28.925932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:28.926011 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:28.926299 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:28.926302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:28.926304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:28.926479 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:28.926580 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:28.926865 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 12:30:28.927027 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":173,"thread_start_us":79,"threads_started":1}
I20260619 12:30:28.927181 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:28.914734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41555 (local address 127.0.0.1:45988)
0619 12:30:28.915195 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:28.915211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:28.915362 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:28.915615 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:28.915618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:28.916036 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:28.916130 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.916136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.916988 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.916994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.917615 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:28.917624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:28.918054 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:28.918057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:28.918204 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:28.921964 (+ 3760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:28.921977 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:28.923497 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:28.925574 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:28.925585 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:28.925591 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:28.925826 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:28.926100 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:28.926105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:28.926107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:28.926189 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:28.926590 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:28.926598 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:28.926703 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 12:30:28.926813 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 12:30:28.926975 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":42,"thread_start_us":122,"threads_started":1}
I20260619 12:30:28.931344 7926 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45988:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 12:30:28.932056 7926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:30:28.937142 8064 tablet_service.cc:1467] Processing CreateTablet for tablet 242c119b5ee3401d813cc7c56e5059cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e06738d952044371bb55a9bed7fc4f95]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:30:28.937234 8204 tablet_service.cc:1467] Processing CreateTablet for tablet 242c119b5ee3401d813cc7c56e5059cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e06738d952044371bb55a9bed7fc4f95]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:30:28.937258 8344 tablet_service.cc:1467] Processing CreateTablet for tablet 242c119b5ee3401d813cc7c56e5059cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e06738d952044371bb55a9bed7fc4f95]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 12:30:28.937530 8064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242c119b5ee3401d813cc7c56e5059cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:28.937572 8344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242c119b5ee3401d813cc7c56e5059cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:28.937613 8204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242c119b5ee3401d813cc7c56e5059cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:28.940300 8440 tablet_bootstrap.cc:492] T 242c119b5ee3401d813cc7c56e5059cc P ff8bd311e16c4523b87c4a29bfdd58d8: Bootstrap starting.
I20260619 12:30:28.940291 8439 tablet_bootstrap.cc:492] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d: Bootstrap starting.
I20260619 12:30:28.940903 8438 tablet_bootstrap.cc:492] T 242c119b5ee3401d813cc7c56e5059cc P 74fb4bf3fe99446a9ea930db56591837: Bootstrap starting.
I20260619 12:30:28.941249 8439 tablet_bootstrap.cc:654] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:28.941249 8440 tablet_bootstrap.cc:654] T 242c119b5ee3401d813cc7c56e5059cc P ff8bd311e16c4523b87c4a29bfdd58d8: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:28.941890 8438 tablet_bootstrap.cc:654] T 242c119b5ee3401d813cc7c56e5059cc P 74fb4bf3fe99446a9ea930db56591837: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:28.943260 8440 tablet_bootstrap.cc:492] T 242c119b5ee3401d813cc7c56e5059cc P ff8bd311e16c4523b87c4a29bfdd58d8: No bootstrap required, opened a new log
I20260619 12:30:28.943320 8439 tablet_bootstrap.cc:492] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d: No bootstrap required, opened a new log
I20260619 12:30:28.943356 8440 ts_tablet_manager.cc:1397] T 242c119b5ee3401d813cc7c56e5059cc P ff8bd311e16c4523b87c4a29bfdd58d8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 12:30:28.943398 8439 ts_tablet_manager.cc:1397] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:30:28.943521 8438 tablet_bootstrap.cc:492] T 242c119b5ee3401d813cc7c56e5059cc P 74fb4bf3fe99446a9ea930db56591837: No bootstrap required, opened a new log
I20260619 12:30:28.943589 8438 ts_tablet_manager.cc:1397] T 242c119b5ee3401d813cc7c56e5059cc P 74fb4bf3fe99446a9ea930db56591837: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 12:30:28.943794 8440 raft_consensus.cc:348] T 242c119b5ee3401d813cc7c56e5059cc P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:28.943899 8440 raft_consensus.cc:374] T 242c119b5ee3401d813cc7c56e5059cc P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:28.943851 8439 raft_consensus.cc:348] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:28.943956 8440 raft_consensus.cc:729] T 242c119b5ee3401d813cc7c56e5059cc P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8bd311e16c4523b87c4a29bfdd58d8, State: Initialized, Role: FOLLOWER
I20260619 12:30:28.943964 8439 raft_consensus.cc:374] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:28.943996 8439 raft_consensus.cc:729] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b066104557584d178b9353215bcae60d, State: Initialized, Role: FOLLOWER
I20260619 12:30:28.944003 8438 raft_consensus.cc:348] T 242c119b5ee3401d813cc7c56e5059cc P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:28.944097 8438 raft_consensus.cc:374] T 242c119b5ee3401d813cc7c56e5059cc P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:28.944077 8440 consensus_queue.cc:260] T 242c119b5ee3401d813cc7c56e5059cc P ff8bd311e16c4523b87c4a29bfdd58d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:28.944131 8438 raft_consensus.cc:729] T 242c119b5ee3401d813cc7c56e5059cc P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fb4bf3fe99446a9ea930db56591837, State: Initialized, Role: FOLLOWER
I20260619 12:30:28.944111 8439 consensus_queue.cc:260] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:28.944238 8438 consensus_queue.cc:260] T 242c119b5ee3401d813cc7c56e5059cc P 74fb4bf3fe99446a9ea930db56591837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:28.944365 8440 ts_tablet_manager.cc:1428] T 242c119b5ee3401d813cc7c56e5059cc P ff8bd311e16c4523b87c4a29bfdd58d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:30:28.944368 8439 ts_tablet_manager.cc:1428] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:30:28.944483 8438 ts_tablet_manager.cc:1428] T 242c119b5ee3401d813cc7c56e5059cc P 74fb4bf3fe99446a9ea930db56591837: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:30:28.944578 8446 tablet_replica.cc:439] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } } }
I20260619 12:30:28.944706 8446 tablet_replica.cc:442] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:30:28.944909 8445 tablet_replica.cc:439] T 242c119b5ee3401d813cc7c56e5059cc P 74fb4bf3fe99446a9ea930db56591837: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } } }
I20260619 12:30:28.945029 8445 tablet_replica.cc:442] T 242c119b5ee3401d813cc7c56e5059cc P 74fb4bf3fe99446a9ea930db56591837: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:30:28.945744 8444 tablet_replica.cc:439] T 242c119b5ee3401d813cc7c56e5059cc P ff8bd311e16c4523b87c4a29bfdd58d8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } } }
I20260619 12:30:28.945909 8444 tablet_replica.cc:442] T 242c119b5ee3401d813cc7c56e5059cc P ff8bd311e16c4523b87c4a29bfdd58d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:30:29.198719 8446 raft_consensus.cc:482] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:30:29.198904 8446 raft_consensus.cc:504] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.199411 8446 leader_election.cc:288] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74fb4bf3fe99446a9ea930db56591837 (127.0.98.2:37411), ff8bd311e16c4523b87c4a29bfdd58d8 (127.0.98.3:42283)
I20260619 12:30:29.199929 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242c119b5ee3401d813cc7c56e5059cc" candidate_uuid: "b066104557584d178b9353215bcae60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fb4bf3fe99446a9ea930db56591837" is_pre_election: true
I20260619 12:30:29.199986 8364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242c119b5ee3401d813cc7c56e5059cc" candidate_uuid: "b066104557584d178b9353215bcae60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" is_pre_election: true
I20260619 12:30:29.200104 8224 raft_consensus.cc:2454] T 242c119b5ee3401d813cc7c56e5059cc P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b066104557584d178b9353215bcae60d in term 0.
I20260619 12:30:29.200170 8364 raft_consensus.cc:2454] T 242c119b5ee3401d813cc7c56e5059cc P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b066104557584d178b9353215bcae60d in term 0.
I20260619 12:30:29.200428 8017 leader_election.cc:302] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74fb4bf3fe99446a9ea930db56591837, b066104557584d178b9353215bcae60d; no voters:
I20260619 12:30:29.200649 8446 raft_consensus.cc:2781] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:30:29.200711 8446 raft_consensus.cc:482] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:30:29.200770 8446 raft_consensus.cc:3037] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:29.201792 8446 raft_consensus.cc:504] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.202116 8446 leader_election.cc:288] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d [CANDIDATE]: Term 1 election: Requested vote from peers 74fb4bf3fe99446a9ea930db56591837 (127.0.98.2:37411), ff8bd311e16c4523b87c4a29bfdd58d8 (127.0.98.3:42283)
I20260619 12:30:29.202491 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242c119b5ee3401d813cc7c56e5059cc" candidate_uuid: "b066104557584d178b9353215bcae60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fb4bf3fe99446a9ea930db56591837"
I20260619 12:30:29.202525 8364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242c119b5ee3401d813cc7c56e5059cc" candidate_uuid: "b066104557584d178b9353215bcae60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8"
I20260619 12:30:29.202611 8224 raft_consensus.cc:3037] T 242c119b5ee3401d813cc7c56e5059cc P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:29.202629 8364 raft_consensus.cc:3037] T 242c119b5ee3401d813cc7c56e5059cc P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:29.203692 8224 raft_consensus.cc:2454] T 242c119b5ee3401d813cc7c56e5059cc P 74fb4bf3fe99446a9ea930db56591837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b066104557584d178b9353215bcae60d in term 1.
I20260619 12:30:29.203692 8364 raft_consensus.cc:2454] T 242c119b5ee3401d813cc7c56e5059cc P ff8bd311e16c4523b87c4a29bfdd58d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b066104557584d178b9353215bcae60d in term 1.
I20260619 12:30:29.204015 8015 leader_election.cc:302] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b066104557584d178b9353215bcae60d, ff8bd311e16c4523b87c4a29bfdd58d8; no voters:
I20260619 12:30:29.204170 8446 raft_consensus.cc:2781] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:29.204288 8446 raft_consensus.cc:686] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d [term 1 LEADER]: Becoming Leader. State: Replica: b066104557584d178b9353215bcae60d, State: Running, Role: LEADER
I20260619 12:30:29.204445 8446 consensus_queue.cc:237] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.205107 8449 tablet_replica.cc:439] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d: TxnStatusTablet state changed. Reason: New leader b066104557584d178b9353215bcae60d. Latest consensus state: current_term: 1 leader_uuid: "b066104557584d178b9353215bcae60d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } } }
I20260619 12:30:29.205230 8449 tablet_replica.cc:442] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:30:29.205652 8483 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 12:30:29.205823 8483 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 12:30:29.206205 7926 catalog_manager.cc:5495] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d reported cstate change: term changed from 0 to 1, leader changed from <none> to b066104557584d178b9353215bcae60d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b066104557584d178b9353215bcae60d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20260619 12:30:29.242854 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:29.239314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40586 (local address 127.0.98.1:41211)
0619 12:30:29.239442 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:29.239446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:29.239495 (+ 49us) server_negotiation.cc:407] Connection header received
0619 12:30:29.239672 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:29.239674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:29.239717 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:29.239783 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:30:29.240052 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.240537 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.241755 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.241977 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.242060 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:29.242423 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:30:29.242494 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:30:29.242561 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:29.242690 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 12:30:29.242737 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:30:29.242892 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:29.239200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41211 (local address 127.0.0.1:40586)
0619 12:30:29.239434 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:29.239449 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:29.239556 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:29.239831 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:29.239834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:29.239842 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:30:29.239949 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:29.239955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.240635 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.240638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:29.241630 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:29.241641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.242085 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.242088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:29.242179 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:29.242231 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:30:29.242584 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:30:29.242597 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 12:30:29.242666 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 12:30:29.242751 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260619 12:30:29.245491 8224 raft_consensus.cc:1261] T 242c119b5ee3401d813cc7c56e5059cc P 74fb4bf3fe99446a9ea930db56591837 [term 1 FOLLOWER]: Refusing update from remote peer b066104557584d178b9353215bcae60d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:29.245491 8364 raft_consensus.cc:1261] T 242c119b5ee3401d813cc7c56e5059cc P ff8bd311e16c4523b87c4a29bfdd58d8 [term 1 FOLLOWER]: Refusing update from remote peer b066104557584d178b9353215bcae60d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:29.245929 8449 consensus_queue.cc:1035] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:29.246065 8446 consensus_queue.cc:1035] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:29.247512 8444 tablet_replica.cc:439] T 242c119b5ee3401d813cc7c56e5059cc P ff8bd311e16c4523b87c4a29bfdd58d8: TxnStatusTablet state changed. Reason: New leader b066104557584d178b9353215bcae60d. Latest consensus state: current_term: 1 leader_uuid: "b066104557584d178b9353215bcae60d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } } }
I20260619 12:30:29.247530 8445 tablet_replica.cc:439] T 242c119b5ee3401d813cc7c56e5059cc P 74fb4bf3fe99446a9ea930db56591837: TxnStatusTablet state changed. Reason: New leader b066104557584d178b9353215bcae60d. Latest consensus state: current_term: 1 leader_uuid: "b066104557584d178b9353215bcae60d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } } }
I20260619 12:30:29.247632 8444 tablet_replica.cc:442] T 242c119b5ee3401d813cc7c56e5059cc P ff8bd311e16c4523b87c4a29bfdd58d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:30:29.247639 8445 tablet_replica.cc:442] T 242c119b5ee3401d813cc7c56e5059cc P 74fb4bf3fe99446a9ea930db56591837: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:30:29.249655 8449 tablet_replica.cc:439] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b066104557584d178b9353215bcae60d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } } }
I20260619 12:30:29.249790 8449 tablet_replica.cc:442] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:30:29.249825 8445 tablet_replica.cc:439] T 242c119b5ee3401d813cc7c56e5059cc P 74fb4bf3fe99446a9ea930db56591837: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b066104557584d178b9353215bcae60d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } } }
I20260619 12:30:29.249918 8445 tablet_replica.cc:442] T 242c119b5ee3401d813cc7c56e5059cc P 74fb4bf3fe99446a9ea930db56591837: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:30:29.249923 8449 tablet_replica.cc:439] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b066104557584d178b9353215bcae60d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } } }
I20260619 12:30:29.250000 8449 tablet_replica.cc:442] T 242c119b5ee3401d813cc7c56e5059cc P b066104557584d178b9353215bcae60d: This TxnStatusTablet replica's current role is: LEADER
I20260619 12:30:29.250640 8453 mvcc.cc:205] Tried to move back new op lower bound from 7298548650985230336 to 7298548650822590464. Current Snapshot: MvccSnapshot[applied={T|T < 7298548650985230336 or (T in {7298548650985230336})}]
I20260619 12:30:29.251037 8452 mvcc.cc:205] Tried to move back new op lower bound from 7298548650985230336 to 7298548650822590464. Current Snapshot: MvccSnapshot[applied={T|T < 7298548650985230336 or (T in {7298548650985230336})}]
I20260619 12:30:29.252668 8444 tablet_replica.cc:439] T 242c119b5ee3401d813cc7c56e5059cc P ff8bd311e16c4523b87c4a29bfdd58d8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b066104557584d178b9353215bcae60d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } } }
I20260619 12:30:29.252786 8444 tablet_replica.cc:442] T 242c119b5ee3401d813cc7c56e5059cc P ff8bd311e16c4523b87c4a29bfdd58d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 12:30:29.259385 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:29.255526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40810 (local address 127.0.98.2:37411)
0619 12:30:29.255633 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:29.255637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:29.255656 (+ 19us) server_negotiation.cc:407] Connection header received
0619 12:30:29.255826 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:29.255828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:29.255890 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:29.255980 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:30:29.256205 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.256741 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.257905 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.258138 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.258283 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:29.258516 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:30:29.258943 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:30:29.259030 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:29.259126 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 12:30:29.259173 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 12:30:29.270644 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:29.267199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37915 (local address 127.0.98.1:41211)
0619 12:30:29.267333 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:29.267338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:29.267556 (+ 218us) server_negotiation.cc:407] Connection header received
0619 12:30:29.267708 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:29.267711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:29.267771 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:29.267848 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:30:29.268105 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.268605 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.269572 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.269808 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.269920 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:29.270203 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:30:29.270284 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:30:29.270364 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:29.270452 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 12:30:29.270506 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:30:29.270676 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:29.267111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41211 (local address 127.0.98.2:37915)
0619 12:30:29.267473 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:29.267485 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:29.267601 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:29.267867 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:29.267870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:29.267877 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:30:29.268000 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:29.268005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.268718 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.268723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:29.269445 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:29.269454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.269919 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.269923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:29.270024 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:29.270065 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:30:29.270368 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:30:29.270376 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 12:30:29.270442 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 12:30:29.270530 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":116,"threads_started":1}
I20260619 12:30:29.277361 8364 raft_consensus.cc:1261] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8 [term 1 FOLLOWER]: Refusing update from remote peer 74fb4bf3fe99446a9ea930db56591837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:29.277573 8084 raft_consensus.cc:1261] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d [term 1 FOLLOWER]: Refusing update from remote peer 74fb4bf3fe99446a9ea930db56591837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:29.278153 8470 consensus_queue.cc:1035] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:29.278347 8445 consensus_queue.cc:1035] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:29.302417 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:29.298270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37411 (local address 127.0.98.1:34183)
0619 12:30:29.298587 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:29.298605 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:29.298729 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:29.298990 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:29.298993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:29.299000 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 12:30:29.299105 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:29.299111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.300057 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.300061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:29.301198 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:29.301209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.301636 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.301639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:29.301724 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:29.301776 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 12:30:29.302107 (+ 331us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 12:30:29.302112 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:30:29.302198 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 12:30:29.302281 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":111,"threads_started":1}
I20260619 12:30:29.302407 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:29.298356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34183 (local address 127.0.98.2:37411)
0619 12:30:29.298524 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:29.298529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:29.298685 (+ 156us) server_negotiation.cc:407] Connection header received
0619 12:30:29.298830 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:29.298833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:29.298901 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:29.298980 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:30:29.299227 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.299957 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.301307 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.301531 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.301614 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:29.301940 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:30:29.302014 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:30:29.302084 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:29.302217 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 12:30:29.302268 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 12:30:29.330928 7926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:29.331117 7926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:29.333712 7926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683: Sending DeleteTablet for 3 replicas of tablet c9830afa7b2f45eb9ea71c9c4b27affa
I20260619 12:30:29.334497 8344 tablet_service.cc:1514] Processing DeleteTablet for tablet c9830afa7b2f45eb9ea71c9c4b27affa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42632
I20260619 12:30:29.334674 8064 tablet_service.cc:1514] Processing DeleteTablet for tablet c9830afa7b2f45eb9ea71c9c4b27affa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40578
I20260619 12:30:29.335244 8204 tablet_service.cc:1514] Processing DeleteTablet for tablet c9830afa7b2f45eb9ea71c9c4b27affa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40798
I20260619 12:30:29.335559 8499 tablet_replica.cc:330] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d: stopping tablet replica
I20260619 12:30:29.335691 8499 raft_consensus.cc:2229] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:29.335834 8499 raft_consensus.cc:2258] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:29.336074 8500 tablet_replica.cc:330] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837: stopping tablet replica
I20260619 12:30:29.336190 8500 raft_consensus.cc:2229] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:30:29.336494 8500 raft_consensus.cc:2258] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:29.336617 8499 ts_tablet_manager.cc:1905] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:29.337165 8500 ts_tablet_manager.cc:1905] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:29.339920 8499 ts_tablet_manager.cc:1918] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:30:29.340013 8499 log.cc:1196] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/wal/wals/c9830afa7b2f45eb9ea71c9c4b27affa
I20260619 12:30:29.340288 8500 ts_tablet_manager.cc:1918] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:30:29.340332 8499 ts_tablet_manager.cc:1939] T c9830afa7b2f45eb9ea71c9c4b27affa P b066104557584d178b9353215bcae60d: Deleting consensus metadata
I20260619 12:30:29.340372 8500 log.cc:1196] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/c9830afa7b2f45eb9ea71c9c4b27affa
I20260619 12:30:29.340642 8500 ts_tablet_manager.cc:1939] T c9830afa7b2f45eb9ea71c9c4b27affa P 74fb4bf3fe99446a9ea930db56591837: Deleting consensus metadata
I20260619 12:30:29.341262 7914 catalog_manager.cc:4841] TS b066104557584d178b9353215bcae60d (127.0.98.1:41211): tablet c9830afa7b2f45eb9ea71c9c4b27affa (table test-table [id=035d01fc54a24bea8af6a176912a8683]) successfully deleted
I20260619 12:30:29.341356 7913 catalog_manager.cc:4841] TS 74fb4bf3fe99446a9ea930db56591837 (127.0.98.2:37411): tablet c9830afa7b2f45eb9ea71c9c4b27affa (table test-table [id=035d01fc54a24bea8af6a176912a8683]) successfully deleted
I20260619 12:30:29.345017 8498 tablet_replica.cc:330] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8: stopping tablet replica
I20260619 12:30:29.345197 8498 raft_consensus.cc:2229] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:29.345342 8498 raft_consensus.cc:2258] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:29.346081 8498 ts_tablet_manager.cc:1905] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:29.348879 8498 ts_tablet_manager.cc:1918] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:30:29.348953 8498 log.cc:1196] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/c9830afa7b2f45eb9ea71c9c4b27affa
I20260619 12:30:29.349231 8498 ts_tablet_manager.cc:1939] T c9830afa7b2f45eb9ea71c9c4b27affa P ff8bd311e16c4523b87c4a29bfdd58d8: Deleting consensus metadata
I20260619 12:30:29.349853 7911 catalog_manager.cc:4841] TS ff8bd311e16c4523b87c4a29bfdd58d8 (127.0.98.3:42283): tablet c9830afa7b2f45eb9ea71c9c4b27affa (table test-table [id=035d01fc54a24bea8af6a176912a8683]) successfully deleted
Jun 19 12:30:29 dist-test-slave-fg3h krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 12:30:29 dist-test-slave-fg3h krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:29 dist-test-slave-fg3h krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 12:30:29.375900 8478 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:29.367552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.0.98.62:41555)
0619 12:30:29.367671 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:29.367675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:29.368164 (+ 489us) server_negotiation.cc:407] Connection header received
0619 12:30:29.368343 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:29.368346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:29.368395 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:29.368447 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:29.369416 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.370252 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.371483 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.371847 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.371916 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:29.375761 (+ 3845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46004: BlockingRecv error: recv got EOF from 127.0.0.1:46004 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 19 12:30:29 dist-test-slave-fg3h krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 12:30:29.388080 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:29.378396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46008 (local address 127.0.98.62:41555)
0619 12:30:29.378534 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:29.378539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:29.378590 (+ 51us) server_negotiation.cc:407] Connection header received
0619 12:30:29.378716 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:29.378719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:29.378775 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:29.378870 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:29.379518 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.380391 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.381286 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.381656 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.381778 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:29.384525 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:29.384530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:29.384533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:29.384561 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:29.386395 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:29.386803 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:29.386807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:29.386809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:29.386899 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:29.387148 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:29.387152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:29.387154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:29.387376 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:29.387517 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:29.387701 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 12:30:29.387904 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 12:30:29.390857 7926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:30:29.391300 7926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:30:29.396272 8064 tablet_service.cc:1467] Processing CreateTablet for tablet 897e9ffff04a4423856d6577aaf306e8 (DEFAULT_TABLE table=test-table [id=3e6bd946bf2f466596d348c6b12dbfc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:29.396407 8344 tablet_service.cc:1467] Processing CreateTablet for tablet 897e9ffff04a4423856d6577aaf306e8 (DEFAULT_TABLE table=test-table [id=3e6bd946bf2f466596d348c6b12dbfc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:29.396569 8064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897e9ffff04a4423856d6577aaf306e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:29.396696 8344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897e9ffff04a4423856d6577aaf306e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:29.397486 8204 tablet_service.cc:1467] Processing CreateTablet for tablet 897e9ffff04a4423856d6577aaf306e8 (DEFAULT_TABLE table=test-table [id=3e6bd946bf2f466596d348c6b12dbfc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:29.397773 8204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 897e9ffff04a4423856d6577aaf306e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:29.398886 8440 tablet_bootstrap.cc:492] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8: Bootstrap starting.
I20260619 12:30:29.399647 8440 tablet_bootstrap.cc:654] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:29.400182 8438 tablet_bootstrap.cc:492] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837: Bootstrap starting.
I20260619 12:30:29.401201 8438 tablet_bootstrap.cc:654] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:29.402159 8440 tablet_bootstrap.cc:492] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8: No bootstrap required, opened a new log
I20260619 12:30:29.402262 8440 ts_tablet_manager.cc:1397] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 12:30:29.402395 8439 tablet_bootstrap.cc:492] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d: Bootstrap starting.
I20260619 12:30:29.402699 8440 raft_consensus.cc:348] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.402810 8440 raft_consensus.cc:374] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:29.402842 8440 raft_consensus.cc:729] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8bd311e16c4523b87c4a29bfdd58d8, State: Initialized, Role: FOLLOWER
I20260619 12:30:29.402972 8440 consensus_queue.cc:260] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.403127 8438 tablet_bootstrap.cc:492] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837: No bootstrap required, opened a new log
I20260619 12:30:29.403198 8439 tablet_bootstrap.cc:654] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:29.403204 8438 ts_tablet_manager.cc:1397] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 12:30:29.403265 8440 ts_tablet_manager.cc:1428] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:30:29.403636 8438 raft_consensus.cc:348] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.403739 8438 raft_consensus.cc:374] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:29.403786 8438 raft_consensus.cc:729] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fb4bf3fe99446a9ea930db56591837, State: Initialized, Role: FOLLOWER
I20260619 12:30:29.403924 8438 consensus_queue.cc:260] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.404194 8438 ts_tablet_manager.cc:1428] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 12:30:29.404728 8439 tablet_bootstrap.cc:492] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d: No bootstrap required, opened a new log
I20260619 12:30:29.404815 8439 ts_tablet_manager.cc:1397] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 12:30:29.405340 8439 raft_consensus.cc:348] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.405452 8439 raft_consensus.cc:374] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:29.405488 8439 raft_consensus.cc:729] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b066104557584d178b9353215bcae60d, State: Initialized, Role: FOLLOWER
I20260619 12:30:29.405599 8439 consensus_queue.cc:260] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.405861 8439 ts_tablet_manager.cc:1428] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:30:29.738586 8446 raft_consensus.cc:482] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:30:29.738730 8446 raft_consensus.cc:504] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.739265 8446 leader_election.cc:288] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74fb4bf3fe99446a9ea930db56591837 (127.0.98.2:37411), ff8bd311e16c4523b87c4a29bfdd58d8 (127.0.98.3:42283)
I20260619 12:30:29.739714 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897e9ffff04a4423856d6577aaf306e8" candidate_uuid: "b066104557584d178b9353215bcae60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fb4bf3fe99446a9ea930db56591837" is_pre_election: true
I20260619 12:30:29.739717 8363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897e9ffff04a4423856d6577aaf306e8" candidate_uuid: "b066104557584d178b9353215bcae60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" is_pre_election: true
I20260619 12:30:29.739884 8224 raft_consensus.cc:2454] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b066104557584d178b9353215bcae60d in term 0.
I20260619 12:30:29.739897 8363 raft_consensus.cc:2454] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b066104557584d178b9353215bcae60d in term 0.
I20260619 12:30:29.740201 8017 leader_election.cc:302] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74fb4bf3fe99446a9ea930db56591837, b066104557584d178b9353215bcae60d; no voters:
I20260619 12:30:29.740391 8446 raft_consensus.cc:2781] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:30:29.740499 8446 raft_consensus.cc:482] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:30:29.740553 8446 raft_consensus.cc:3037] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:29.741612 8446 raft_consensus.cc:504] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.741997 8446 leader_election.cc:288] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [CANDIDATE]: Term 1 election: Requested vote from peers 74fb4bf3fe99446a9ea930db56591837 (127.0.98.2:37411), ff8bd311e16c4523b87c4a29bfdd58d8 (127.0.98.3:42283)
I20260619 12:30:29.742338 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897e9ffff04a4423856d6577aaf306e8" candidate_uuid: "b066104557584d178b9353215bcae60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fb4bf3fe99446a9ea930db56591837"
I20260619 12:30:29.742395 8363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "897e9ffff04a4423856d6577aaf306e8" candidate_uuid: "b066104557584d178b9353215bcae60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8"
I20260619 12:30:29.742456 8224 raft_consensus.cc:3037] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:29.742497 8363 raft_consensus.cc:3037] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:29.743575 8224 raft_consensus.cc:2454] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b066104557584d178b9353215bcae60d in term 1.
I20260619 12:30:29.743667 8363 raft_consensus.cc:2454] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b066104557584d178b9353215bcae60d in term 1.
I20260619 12:30:29.743935 8017 leader_election.cc:302] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 74fb4bf3fe99446a9ea930db56591837, b066104557584d178b9353215bcae60d, ff8bd311e16c4523b87c4a29bfdd58d8; no voters:
I20260619 12:30:29.744143 8446 raft_consensus.cc:2781] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:29.744235 8446 raft_consensus.cc:686] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [term 1 LEADER]: Becoming Leader. State: Replica: b066104557584d178b9353215bcae60d, State: Running, Role: LEADER
I20260619 12:30:29.744378 8446 consensus_queue.cc:237] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.746358 7923 catalog_manager.cc:5495] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d reported cstate change: term changed from 0 to 1, leader changed from <none> to b066104557584d178b9353215bcae60d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b066104557584d178b9353215bcae60d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20260619 12:30:29.768687 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:29.765722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.0.98.1:41211)
0619 12:30:29.765843 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:29.765846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:29.765860 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:30:29.765977 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:29.765980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:29.766023 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:29.766095 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:30:29.766334 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.766852 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.767684 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.767905 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.767987 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:29.768248 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:30:29.768321 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:30:29.768396 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:29.768538 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 12:30:29.768584 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 12:30:29.770628 8363 raft_consensus.cc:1261] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8 [term 1 FOLLOWER]: Refusing update from remote peer b066104557584d178b9353215bcae60d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:29.770668 8224 raft_consensus.cc:1261] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837 [term 1 FOLLOWER]: Refusing update from remote peer b066104557584d178b9353215bcae60d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:29.771041 8446 consensus_queue.cc:1035] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:29.771199 8457 consensus_queue.cc:1035] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:29.778020 7926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:29.778168 7926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:29.780213 7926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683: Sending DeleteTablet for 3 replicas of tablet 897e9ffff04a4423856d6577aaf306e8
I20260619 12:30:29.780692 8064 tablet_service.cc:1514] Processing DeleteTablet for tablet 897e9ffff04a4423856d6577aaf306e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40578
I20260619 12:30:29.780865 8499 tablet_replica.cc:330] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d: stopping tablet replica
I20260619 12:30:29.780882 8344 tablet_service.cc:1514] Processing DeleteTablet for tablet 897e9ffff04a4423856d6577aaf306e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42632
I20260619 12:30:29.780974 8204 tablet_service.cc:1514] Processing DeleteTablet for tablet 897e9ffff04a4423856d6577aaf306e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40798
I20260619 12:30:29.781013 8499 raft_consensus.cc:2229] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:30:29.781047 8498 tablet_replica.cc:330] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8: stopping tablet replica
I20260619 12:30:29.781181 8498 raft_consensus.cc:2229] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:29.781217 8499 raft_consensus.cc:2258] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:29.781260 8500 tablet_replica.cc:330] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837: stopping tablet replica
I20260619 12:30:29.781330 8498 raft_consensus.cc:2258] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:29.781365 8500 raft_consensus.cc:2229] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:29.781513 8500 raft_consensus.cc:2258] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:29.781191 8343 tablet_service.cc:1514] Processing DeleteTablet for tablet 897e9ffff04a4423856d6577aaf306e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42632
I20260619 12:30:29.782083 8498 ts_tablet_manager.cc:1905] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:29.782335 8500 ts_tablet_manager.cc:1905] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:29.782589 8499 ts_tablet_manager.cc:1905] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:29.782733 7926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:30:29.783134 7926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:30:29.784670 8500 ts_tablet_manager.cc:1918] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:30:29.784742 8500 log.cc:1196] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/897e9ffff04a4423856d6577aaf306e8
I20260619 12:30:29.784727 8498 ts_tablet_manager.cc:1918] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:30:29.784794 8498 log.cc:1196] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/897e9ffff04a4423856d6577aaf306e8
I20260619 12:30:29.785022 8500 ts_tablet_manager.cc:1939] T 897e9ffff04a4423856d6577aaf306e8 P 74fb4bf3fe99446a9ea930db56591837: Deleting consensus metadata
I20260619 12:30:29.785043 8498 ts_tablet_manager.cc:1939] T 897e9ffff04a4423856d6577aaf306e8 P ff8bd311e16c4523b87c4a29bfdd58d8: Deleting consensus metadata
I20260619 12:30:29.785099 8499 ts_tablet_manager.cc:1918] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:30:29.785172 8499 log.cc:1196] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/wal/wals/897e9ffff04a4423856d6577aaf306e8
I20260619 12:30:29.786289 7913 catalog_manager.cc:4841] TS 74fb4bf3fe99446a9ea930db56591837 (127.0.98.2:37411): tablet 897e9ffff04a4423856d6577aaf306e8 (table test-table [id=3e6bd946bf2f466596d348c6b12dbfc3]) successfully deleted
I20260619 12:30:29.786366 7911 catalog_manager.cc:4841] TS ff8bd311e16c4523b87c4a29bfdd58d8 (127.0.98.3:42283): tablet 897e9ffff04a4423856d6577aaf306e8 (table test-table [id=3e6bd946bf2f466596d348c6b12dbfc3]) successfully deleted
I20260619 12:30:29.786482 8499 ts_tablet_manager.cc:1939] T 897e9ffff04a4423856d6577aaf306e8 P b066104557584d178b9353215bcae60d: Deleting consensus metadata
W20260619 12:30:29.786603 7911 catalog_manager.cc:4808] TS ff8bd311e16c4523b87c4a29bfdd58d8 (127.0.98.3:42283): delete failed for tablet 897e9ffff04a4423856d6577aaf306e8 because the tablet was not found. No further retry: Not found: Tablet not found: 897e9ffff04a4423856d6577aaf306e8
I20260619 12:30:29.787124 7914 catalog_manager.cc:4841] TS b066104557584d178b9353215bcae60d (127.0.98.1:41211): tablet 897e9ffff04a4423856d6577aaf306e8 (table test-table [id=3e6bd946bf2f466596d348c6b12dbfc3]) successfully deleted
I20260619 12:30:29.787882 8064 tablet_service.cc:1467] Processing CreateTablet for tablet 5974f567d3834da4817e23a652bf0bb7 (DEFAULT_TABLE table=test-table [id=e0a96ad84c22494bb217ab04c061f579]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:29.788065 8343 tablet_service.cc:1467] Processing CreateTablet for tablet 5974f567d3834da4817e23a652bf0bb7 (DEFAULT_TABLE table=test-table [id=e0a96ad84c22494bb217ab04c061f579]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:29.788134 8064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5974f567d3834da4817e23a652bf0bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:29.788118 8204 tablet_service.cc:1467] Processing CreateTablet for tablet 5974f567d3834da4817e23a652bf0bb7 (DEFAULT_TABLE table=test-table [id=e0a96ad84c22494bb217ab04c061f579]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:29.788319 8204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5974f567d3834da4817e23a652bf0bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:29.788338 8343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5974f567d3834da4817e23a652bf0bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:29.789968 8439 tablet_bootstrap.cc:492] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d: Bootstrap starting.
I20260619 12:30:29.790498 8440 tablet_bootstrap.cc:492] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8: Bootstrap starting.
I20260619 12:30:29.790548 8438 tablet_bootstrap.cc:492] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837: Bootstrap starting.
I20260619 12:30:29.790757 8439 tablet_bootstrap.cc:654] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:29.791330 8438 tablet_bootstrap.cc:654] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:29.791412 8440 tablet_bootstrap.cc:654] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:29.792403 8438 tablet_bootstrap.cc:492] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837: No bootstrap required, opened a new log
I20260619 12:30:29.792493 8438 ts_tablet_manager.cc:1397] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 12:30:29.792570 8439 tablet_bootstrap.cc:492] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d: No bootstrap required, opened a new log
I20260619 12:30:29.792649 8439 ts_tablet_manager.cc:1397] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 12:30:29.792904 8438 raft_consensus.cc:348] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.793023 8438 raft_consensus.cc:374] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:29.793059 8438 raft_consensus.cc:729] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fb4bf3fe99446a9ea930db56591837, State: Initialized, Role: FOLLOWER
I20260619 12:30:29.793004 8439 raft_consensus.cc:348] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.793154 8439 raft_consensus.cc:374] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:29.793197 8439 raft_consensus.cc:729] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b066104557584d178b9353215bcae60d, State: Initialized, Role: FOLLOWER
I20260619 12:30:29.793219 8438 consensus_queue.cc:260] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.793313 8439 consensus_queue.cc:260] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.793495 8438 ts_tablet_manager.cc:1428] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:30:29.793587 8439 ts_tablet_manager.cc:1428] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 12:30:29.794021 8440 tablet_bootstrap.cc:492] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8: No bootstrap required, opened a new log
I20260619 12:30:29.794121 8440 ts_tablet_manager.cc:1397] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260619 12:30:29.794538 8440 raft_consensus.cc:348] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.794674 8440 raft_consensus.cc:374] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:29.794708 8440 raft_consensus.cc:729] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8bd311e16c4523b87c4a29bfdd58d8, State: Initialized, Role: FOLLOWER
I20260619 12:30:29.794836 8440 consensus_queue.cc:260] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.795156 8440 ts_tablet_manager.cc:1428] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260619 12:30:29.874217 8495 raft_consensus.cc:482] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:30:29.874387 8495 raft_consensus.cc:504] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.874826 8495 leader_election.cc:288] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b066104557584d178b9353215bcae60d (127.0.98.1:41211), ff8bd311e16c4523b87c4a29bfdd58d8 (127.0.98.3:42283)
I20260619 12:30:29.875299 8084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5974f567d3834da4817e23a652bf0bb7" candidate_uuid: "74fb4bf3fe99446a9ea930db56591837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b066104557584d178b9353215bcae60d" is_pre_election: true
I20260619 12:30:29.875469 8084 raft_consensus.cc:2454] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74fb4bf3fe99446a9ea930db56591837 in term 0.
I20260619 12:30:29.875346 8363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5974f567d3834da4817e23a652bf0bb7" candidate_uuid: "74fb4bf3fe99446a9ea930db56591837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" is_pre_election: true
I20260619 12:30:29.875689 8363 raft_consensus.cc:2454] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74fb4bf3fe99446a9ea930db56591837 in term 0.
I20260619 12:30:29.875795 8158 leader_election.cc:302] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74fb4bf3fe99446a9ea930db56591837, b066104557584d178b9353215bcae60d; no voters:
I20260619 12:30:29.875989 8495 raft_consensus.cc:2781] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:30:29.876053 8495 raft_consensus.cc:482] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:30:29.876102 8495 raft_consensus.cc:3037] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:29.877286 8495 raft_consensus.cc:504] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.877698 8495 leader_election.cc:288] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [CANDIDATE]: Term 1 election: Requested vote from peers b066104557584d178b9353215bcae60d (127.0.98.1:41211), ff8bd311e16c4523b87c4a29bfdd58d8 (127.0.98.3:42283)
I20260619 12:30:29.878000 8084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5974f567d3834da4817e23a652bf0bb7" candidate_uuid: "74fb4bf3fe99446a9ea930db56591837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b066104557584d178b9353215bcae60d"
I20260619 12:30:29.878141 8084 raft_consensus.cc:3037] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:29.878149 8363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5974f567d3834da4817e23a652bf0bb7" candidate_uuid: "74fb4bf3fe99446a9ea930db56591837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8"
I20260619 12:30:29.878276 8363 raft_consensus.cc:3037] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:29.879105 8084 raft_consensus.cc:2454] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74fb4bf3fe99446a9ea930db56591837 in term 1.
I20260619 12:30:29.879184 8363 raft_consensus.cc:2454] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74fb4bf3fe99446a9ea930db56591837 in term 1.
I20260619 12:30:29.879405 8158 leader_election.cc:302] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74fb4bf3fe99446a9ea930db56591837, b066104557584d178b9353215bcae60d; no voters:
I20260619 12:30:29.879603 8495 raft_consensus.cc:2781] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:29.879758 8495 raft_consensus.cc:686] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [term 1 LEADER]: Becoming Leader. State: Replica: 74fb4bf3fe99446a9ea930db56591837, State: Running, Role: LEADER
I20260619 12:30:29.879904 8495 consensus_queue.cc:237] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } }
I20260619 12:30:29.881596 7927 catalog_manager.cc:5495] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74fb4bf3fe99446a9ea930db56591837 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "74fb4bf3fe99446a9ea930db56591837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74fb4bf3fe99446a9ea930db56591837" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20260619 12:30:29.899834 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:29.896620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40822 (local address 127.0.98.2:37411)
0619 12:30:29.896832 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:29.896841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:29.896852 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:30:29.896890 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:29.896893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:29.896932 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:29.897034 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:30:29.897273 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.897884 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.898734 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:29.898979 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:29.899087 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:29.899376 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:30:29.899461 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:30:29.899545 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:29.899631 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 12:30:29.899682 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260619 12:30:29.905920 8363 raft_consensus.cc:1261] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8 [term 1 FOLLOWER]: Refusing update from remote peer 74fb4bf3fe99446a9ea930db56591837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:29.905949 8084 raft_consensus.cc:1261] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d [term 1 FOLLOWER]: Refusing update from remote peer 74fb4bf3fe99446a9ea930db56591837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:29.906313 8495 consensus_queue.cc:1035] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b066104557584d178b9353215bcae60d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:29.906464 8524 consensus_queue.cc:1035] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8bd311e16c4523b87c4a29bfdd58d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:29.942289 7926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:29.942451 7926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:29.944387 7926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8164e8dab6e456985235b59053ff683: Sending DeleteTablet for 3 replicas of tablet 5974f567d3834da4817e23a652bf0bb7
I20260619 12:30:29.944990 8204 tablet_service.cc:1514] Processing DeleteTablet for tablet 5974f567d3834da4817e23a652bf0bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40798
I20260619 12:30:29.945140 8064 tablet_service.cc:1514] Processing DeleteTablet for tablet 5974f567d3834da4817e23a652bf0bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40578
I20260619 12:30:29.945225 8500 tablet_replica.cc:330] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837: stopping tablet replica
I20260619 12:30:29.945214 8343 tablet_service.cc:1514] Processing DeleteTablet for tablet 5974f567d3834da4817e23a652bf0bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:29 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42632
I20260619 12:30:29.945333 8499 tablet_replica.cc:330] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d: stopping tablet replica
I20260619 12:30:29.945367 8500 raft_consensus.cc:2229] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:30:29.945417 8498 tablet_replica.cc:330] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8: stopping tablet replica
I20260619 12:30:29.945456 8499 raft_consensus.cc:2229] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:29.945557 8500 raft_consensus.cc:2258] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:29.945557 8498 raft_consensus.cc:2229] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:29.945586 8499 raft_consensus.cc:2258] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:29.945664 8498 raft_consensus.cc:2258] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:29.946409 8499 ts_tablet_manager.cc:1905] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:29.946408 8498 ts_tablet_manager.cc:1905] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:29.946903 8500 ts_tablet_manager.cc:1905] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:29.948892 8500 ts_tablet_manager.cc:1918] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:30:29.948992 8500 log.cc:1196] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/5974f567d3834da4817e23a652bf0bb7
I20260619 12:30:29.948984 8498 ts_tablet_manager.cc:1918] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:30:29.949054 8498 log.cc:1196] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/5974f567d3834da4817e23a652bf0bb7
I20260619 12:30:29.949050 8499 ts_tablet_manager.cc:1918] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 12:30:29.949148 8499 log.cc:1196] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781872188319340-392-0/minicluster-data/ts-0/wal/wals/5974f567d3834da4817e23a652bf0bb7
I20260619 12:30:29.949290 8500 ts_tablet_manager.cc:1939] T 5974f567d3834da4817e23a652bf0bb7 P 74fb4bf3fe99446a9ea930db56591837: Deleting consensus metadata
I20260619 12:30:29.949357 8498 ts_tablet_manager.cc:1939] T 5974f567d3834da4817e23a652bf0bb7 P ff8bd311e16c4523b87c4a29bfdd58d8: Deleting consensus metadata
I20260619 12:30:29.949460 8499 ts_tablet_manager.cc:1939] T 5974f567d3834da4817e23a652bf0bb7 P b066104557584d178b9353215bcae60d: Deleting consensus metadata
I20260619 12:30:29.950033 7913 catalog_manager.cc:4841] TS 74fb4bf3fe99446a9ea930db56591837 (127.0.98.2:37411): tablet 5974f567d3834da4817e23a652bf0bb7 (table test-table [id=e0a96ad84c22494bb217ab04c061f579]) successfully deleted
I20260619 12:30:29.950028 7914 catalog_manager.cc:4841] TS b066104557584d178b9353215bcae60d (127.0.98.1:41211): tablet 5974f567d3834da4817e23a652bf0bb7 (table test-table [id=e0a96ad84c22494bb217ab04c061f579]) successfully deleted
I20260619 12:30:29.950196 7911 catalog_manager.cc:4841] TS ff8bd311e16c4523b87c4a29bfdd58d8 (127.0.98.3:42283): tablet 5974f567d3834da4817e23a652bf0bb7 (table test-table [id=e0a96ad84c22494bb217ab04c061f579]) successfully deleted
I20260619 12:30:29.950609 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 7999
I20260619 12:30:29.958307 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 8139
I20260619 12:30:29.965013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 8279
I20260619 12:30:29.970664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 7895
2026-06-19T12:30:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3730 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:30:30 dist-test-slave-fg3h krb5kdc[8535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:30:30 dist-test-slave-fg3h krb5kdc[8535](info): set up 2 sockets
Jun 19 12:30:30 dist-test-slave-fg3h krb5kdc[8535](info): commencing operation
krb5kdc: starting...
W20260619 12:30:31.013908 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:30:31 dist-test-slave-fg3h krb5kdc[8535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:30:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:30:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:30:31.139959 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45955
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:31.248209 8548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:31.248469 8548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:31.248509 8548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:31.252538 8548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:31.252599 8548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:31.252619 8548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:31.252635 8548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:31.252650 8548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:30:31.257962 8548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:31.258996 8548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:31.260012 8548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:31.268724 8554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:31.268715 8556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:31.268713 8553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:31.269017 8548 server_base.cc:1034] running on GCE node
I20260619 12:30:31.269801 8548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:31.271430 8548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:31.272668 8548 hybrid_clock.cc:648] HybridClock initialized: now 1781872231272650 us; error 45 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T12:30:31Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1305 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:30:31 dist-test-slave-fg3h krb5kdc[8561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:30:31 dist-test-slave-fg3h krb5kdc[8561](info): set up 2 sockets
Jun 19 12:30:31 dist-test-slave-fg3h krb5kdc[8561](info): commencing operation
krb5kdc: starting...
W20260619 12:30:32.329835 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:30:32 dist-test-slave-fg3h krb5kdc[8561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:30:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:30:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:30:32.456111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43419
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:32.566792 8574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:32.567090 8574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:32.567171 8574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:32.571147 8574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:32.571249 8574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:32.571286 8574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:32.571322 8574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:32.571352 8574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:30:32.576622 8574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:32.577706 8574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:32.578727 8574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:32.589524 8582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:32.589628 8574 server_base.cc:1034] running on GCE node
W20260619 12:30:32.589535 8580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:32.589535 8579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:32.590348 8574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:32.591410 8574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:32.592594 8574 hybrid_clock.cc:648] HybridClock initialized: now 1781872232592562 us; error 47 us; skew 500 ppm
Jun 19 12:30:32 dist-test-slave-fg3h krb5kdc[8561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:32.595374 8574 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:30:32.596171 8574 webserver.cc:460] Webserver started at http://127.0.98.62:42201/ using document root <none> and password file <none>
I20260619 12:30:32.596689 8574 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:32.596751 8574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:32.596910 8574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:32.598953 8574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "f225e2a39e98498aaab40497179a00ba"
format_stamp: "Formatted at 2026-06-19 12:30:32 on dist-test-slave-fg3h"
server_key: "e8466d2b120e00c13cf90854156c65da"
server_key_iv: "3f7f2c02cce69a847f3f2973e8d8263a"
server_key_version: "encryptionkey@0"
I20260619 12:30:32.599478 8574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "f225e2a39e98498aaab40497179a00ba"
format_stamp: "Formatted at 2026-06-19 12:30:32 on dist-test-slave-fg3h"
server_key: "e8466d2b120e00c13cf90854156c65da"
server_key_iv: "3f7f2c02cce69a847f3f2973e8d8263a"
server_key_version: "encryptionkey@0"
I20260619 12:30:32.603209 8574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:30:32.605667 8589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:32.607029 8574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:32.607172 8574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "f225e2a39e98498aaab40497179a00ba"
format_stamp: "Formatted at 2026-06-19 12:30:32 on dist-test-slave-fg3h"
server_key: "e8466d2b120e00c13cf90854156c65da"
server_key_iv: "3f7f2c02cce69a847f3f2973e8d8263a"
server_key_version: "encryptionkey@0"
I20260619 12:30:32.607283 8574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:32.620024 8574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:32.623473 8574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:32.623680 8574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:32.630543 8574 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43419
I20260619 12:30:32.630618 8641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43419 every 8 connection(s)
I20260619 12:30:32.631520 8574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:30:32.632510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 8574
I20260619 12:30:32.632622 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:30:32.632933 392 external_mini_cluster.cc:1428] Setting key c26c470138242aeb16d3227e3f464ff0
I20260619 12:30:32.634938 8642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:32.641546 8642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba: Bootstrap starting.
Jun 19 12:30:32 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:32.644219 8642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:32.644960 8642 log.cc:824] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:32.646860 8642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba: No bootstrap required, opened a new log
I20260619 12:30:32.648423 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:32.634650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43708 (local address 127.0.98.62:43419)
0619 12:30:32.635238 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:32.635254 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:32.635329 (+ 75us) server_negotiation.cc:407] Connection header received
0619 12:30:32.636101 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:32.636118 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:32.636460 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:32.636792 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:32.637722 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:32.638741 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:32.639904 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:32.640375 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:32.640502 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:32.643408 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:32.643413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:32.643426 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:32.643460 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:32.645406 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:32.645851 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:32.645861 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:32.645865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:32.645972 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:32.646264 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:32.646267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:32.646269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:32.646671 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:32.646879 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:32.647144 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 12:30:32.647468 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":147,"threads_started":1}
I20260619 12:30:32.651111 8642 raft_consensus.cc:348] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f225e2a39e98498aaab40497179a00ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43419 } }
I20260619 12:30:32.651329 8642 raft_consensus.cc:374] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:32.651363 8642 raft_consensus.cc:729] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f225e2a39e98498aaab40497179a00ba, State: Initialized, Role: FOLLOWER
I20260619 12:30:32.651757 8642 consensus_queue.cc:260] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f225e2a39e98498aaab40497179a00ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43419 } }
I20260619 12:30:32.651876 8642 raft_consensus.cc:388] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:30:32.651958 8642 raft_consensus.cc:482] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:30:32.652056 8642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:32.652968 8642 raft_consensus.cc:504] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f225e2a39e98498aaab40497179a00ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43419 } }
I20260619 12:30:32.653334 8642 leader_election.cc:302] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f225e2a39e98498aaab40497179a00ba; no voters:
I20260619 12:30:32.653683 8642 leader_election.cc:288] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:30:32.653857 8647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:32.654224 8647 raft_consensus.cc:686] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [term 1 LEADER]: Becoming Leader. State: Replica: f225e2a39e98498aaab40497179a00ba, State: Running, Role: LEADER
I20260619 12:30:32.654560 8647 consensus_queue.cc:237] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f225e2a39e98498aaab40497179a00ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43419 } }
I20260619 12:30:32.654878 8642 sys_catalog.cc:564] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:32.656388 8648 sys_catalog.cc:455] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f225e2a39e98498aaab40497179a00ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f225e2a39e98498aaab40497179a00ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43419 } } }
I20260619 12:30:32.656412 8649 sys_catalog.cc:455] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader f225e2a39e98498aaab40497179a00ba. Latest consensus state: current_term: 1 leader_uuid: "f225e2a39e98498aaab40497179a00ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f225e2a39e98498aaab40497179a00ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43419 } } }
I20260619 12:30:32.656562 8649 sys_catalog.cc:458] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:32.656562 8648 sys_catalog.cc:458] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [sys.catalog]: This master's current role is: LEADER
W20260619 12:30:32.660255 8662 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:30:32.660342 8662 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:30:32.660422 8661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:30:32.661406 8661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:30:32.667197 8661 catalog_manager.cc:1347] Generated new cluster ID: f4d7aae1747b4cff80d288b9a33479eb
I20260619 12:30:32.667273 8661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:30:32.676316 8661 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:30:32.677657 8661 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:30:32.683578 8661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba: Generated new TSK 0
I20260619 12:30:32.684437 8661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:30:32.720956 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43419
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:32.831375 8668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:32.831665 8668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:32.831736 8668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:32.835978 8668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:32.836079 8668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:32.836171 8668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:30:32.841529 8668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:32.842628 8668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:32.843675 8668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:30:32.853479 8668 server_base.cc:1034] running on GCE node
W20260619 12:30:32.853500 8676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:32.853405 8673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:32.853449 8674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:32.854128 8668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:32.854766 8668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:32.855991 8668 hybrid_clock.cc:648] HybridClock initialized: now 1781872232855942 us; error 77 us; skew 500 ppm
Jun 19 12:30:32 dist-test-slave-fg3h krb5kdc[8561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:32.859017 8668 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:30:32.859956 8668 webserver.cc:460] Webserver started at http://127.0.98.1:33621/ using document root <none> and password file <none>
I20260619 12:30:32.860534 8668 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:32.860630 8668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:32.860865 8668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:32.863193 8668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "55f3a0c52c884ce89f3ef462e877618d"
format_stamp: "Formatted at 2026-06-19 12:30:32 on dist-test-slave-fg3h"
server_key: "24c8908150e7e59ff1d67fd9f32396f5"
server_key_iv: "05593b1e6e7d24d1f9d4e2e182d585cc"
server_key_version: "encryptionkey@0"
I20260619 12:30:32.863966 8668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "55f3a0c52c884ce89f3ef462e877618d"
format_stamp: "Formatted at 2026-06-19 12:30:32 on dist-test-slave-fg3h"
server_key: "24c8908150e7e59ff1d67fd9f32396f5"
server_key_iv: "05593b1e6e7d24d1f9d4e2e182d585cc"
server_key_version: "encryptionkey@0"
I20260619 12:30:32.867965 8668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:30:32.870535 8683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:32.871851 8668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:30:32.871999 8668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "55f3a0c52c884ce89f3ef462e877618d"
format_stamp: "Formatted at 2026-06-19 12:30:32 on dist-test-slave-fg3h"
server_key: "24c8908150e7e59ff1d67fd9f32396f5"
server_key_iv: "05593b1e6e7d24d1f9d4e2e182d585cc"
server_key_version: "encryptionkey@0"
I20260619 12:30:32.872121 8668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:32.884330 8668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:32.887288 8668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:32.887485 8668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:32.888078 8668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:32.888877 8668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:32.888947 8668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:32.889016 8668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:32.889065 8668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:32.897634 8668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46235
I20260619 12:30:32.897653 8796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46235 every 8 connection(s)
I20260619 12:30:32.898492 8668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:30:32.907508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 8668
I20260619 12:30:32.907651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:30:32.907989 392 external_mini_cluster.cc:1428] Setting key 0ee2baab7acdcfb5dbfc55f3d909bcdf
Jun 19 12:30:32 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:32.913995 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:32.900484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33813 (local address 127.0.98.62:43419)
0619 12:30:32.900653 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:32.900657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:32.901448 (+ 791us) server_negotiation.cc:407] Connection header received
0619 12:30:32.902813 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:32.902816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:32.902861 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:32.902944 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:32.904122 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:32.904660 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:32.905846 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:32.906069 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:32.906156 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:32.908982 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:32.908985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:32.908987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:32.909009 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:32.911669 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:32.912207 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:32.912212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:32.912214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:32.912303 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:32.912728 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:32.912735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:32.912739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:32.912946 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:32.913031 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:32.913698 (+ 667us) server_negotiation.cc:299] Negotiation successful
0619 12:30:32.913835 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 12:30:32.915164 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:32.900836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.1:33813)
0619 12:30:32.901299 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:32.901340 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:32.902607 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:32.903137 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:32.903143 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:32.903501 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:32.903971 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:32.903994 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:32.904808 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:32.904811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:32.905730 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:32.905740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:32.906205 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:32.906208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:32.906343 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:32.907085 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:32.907102 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:32.908836 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:32.911804 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:32.911810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:32.911825 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:32.912098 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:32.912405 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:32.912411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:32.912415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:32.912619 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:32.913060 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:32.913070 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:32.913419 (+ 349us) client_negotiation.cc:769] Sending connection context
0619 12:30:32.913652 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 12:30:32.914022 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":120,"threads_started":1}
I20260619 12:30:32.916404 8797 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43419
I20260619 12:30:32.916677 8797 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:32.917296 8797 heartbeater.cc:511] Master 127.0.98.62:43419 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:30:32.918893 8606 ts_manager.cc:194] Registered new tserver with Master: 55f3a0c52c884ce89f3ef462e877618d (127.0.98.1:46235)
I20260619 12:30:32.920032 8606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:30:32.937237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43419
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:33.046404 8802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:33.046700 8802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:33.046746 8802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:33.051010 8802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:33.051072 8802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:33.051131 8802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:30:33.056592 8802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:33.057628 8802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:33.058579 8802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:33.068641 8810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:33.068674 8807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:33.068671 8808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:33.068851 8802 server_base.cc:1034] running on GCE node
I20260619 12:30:33.069433 8802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:33.070056 8802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:33.071241 8802 hybrid_clock.cc:648] HybridClock initialized: now 1781872233071207 us; error 49 us; skew 500 ppm
Jun 19 12:30:33 dist-test-slave-fg3h krb5kdc[8561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:33.074172 8802 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:30:33.074991 8802 webserver.cc:460] Webserver started at http://127.0.98.2:40173/ using document root <none> and password file <none>
I20260619 12:30:33.075593 8802 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:33.075654 8802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:33.075884 8802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:33.078069 8802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "f73a4f5dfa85470ba1d20fa4ce0222d0"
format_stamp: "Formatted at 2026-06-19 12:30:33 on dist-test-slave-fg3h"
server_key: "932896148af020025141ccdc3300b37a"
server_key_iv: "db6f71a45392f0881332ce1a80f47025"
server_key_version: "encryptionkey@0"
I20260619 12:30:33.078593 8802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f73a4f5dfa85470ba1d20fa4ce0222d0"
format_stamp: "Formatted at 2026-06-19 12:30:33 on dist-test-slave-fg3h"
server_key: "932896148af020025141ccdc3300b37a"
server_key_iv: "db6f71a45392f0881332ce1a80f47025"
server_key_version: "encryptionkey@0"
I20260619 12:30:33.082406 8802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 12:30:33.084928 8817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:33.086215 8802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:33.086313 8802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "f73a4f5dfa85470ba1d20fa4ce0222d0"
format_stamp: "Formatted at 2026-06-19 12:30:33 on dist-test-slave-fg3h"
server_key: "932896148af020025141ccdc3300b37a"
server_key_iv: "db6f71a45392f0881332ce1a80f47025"
server_key_version: "encryptionkey@0"
I20260619 12:30:33.086400 8802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:33.102214 8802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:33.105683 8802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:33.105901 8802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:33.106539 8802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:33.107399 8802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:33.107446 8802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:33.107549 8802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:33.107584 8802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:33.116430 8802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33125
I20260619 12:30:33.116452 8930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33125 every 8 connection(s)
I20260619 12:30:33.117479 8802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:30:33.123521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 8802
I20260619 12:30:33.123667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:30:33.123960 392 external_mini_cluster.cc:1428] Setting key b902bc3ea0da0a287b6be6f6192a9950
Jun 19 12:30:33 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:30:33.132678 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:33.119501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57877 (local address 127.0.98.62:43419)
0619 12:30:33.119677 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:33.119681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:33.120974 (+ 1293us) server_negotiation.cc:407] Connection header received
0619 12:30:33.122018 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:33.122021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:33.122075 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:33.122150 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:33.123716 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.124425 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.125900 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.126153 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.126280 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:33.129136 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:33.129140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:33.129142 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:33.129174 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:33.130682 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:33.131189 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:33.131194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:33.131196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:33.131275 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:33.131618 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:33.131623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:33.131624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:33.131800 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:33.131902 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:33.132404 (+ 502us) server_negotiation.cc:299] Negotiation successful
0619 12:30:33.132519 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 12:30:33.133667 8933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:33.119916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.2:57877)
0619 12:30:33.120807 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:33.120849 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:33.121797 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:33.122487 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:33.122495 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:33.122865 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:33.123437 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:33.123458 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.124605 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.124608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:33.125765 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:33.125775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.126318 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.126322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:33.126522 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:33.127185 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:33.127203 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:33.128924 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:33.130814 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:33.130819 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:33.130833 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:33.131089 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:33.131369 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:33.131372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:33.131374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:33.131515 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:33.131910 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:33.131919 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:33.132167 (+ 248us) client_negotiation.cc:769] Sending connection context
0619 12:30:33.132350 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 12:30:33.132689 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":549,"threads_started":1}
I20260619 12:30:33.134826 8931 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 12:30:33.135069 8931 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:33.135622 8931 heartbeater.cc:511] Master 127.0.98.62:43419 requested a full tablet report, sending...
I20260619 12:30:33.136777 8606 ts_manager.cc:194] Registered new tserver with Master: f73a4f5dfa85470ba1d20fa4ce0222d0 (127.0.98.2:33125)
I20260619 12:30:33.137626 8606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:30:33.152894 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43419
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:33.263115 8936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:33.263398 8936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:33.263473 8936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:33.267800 8936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:33.267889 8936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:33.267971 8936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:30:33.273290 8936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:33.274312 8936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:33.275364 8936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:30:33.285050 8936 server_base.cc:1034] running on GCE node
W20260619 12:30:33.284992 8944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:33.285012 8941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:33.284988 8942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:33.285707 8936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:33.286273 8936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:33.287488 8936 hybrid_clock.cc:648] HybridClock initialized: now 1781872233287466 us; error 49 us; skew 500 ppm
Jun 19 12:30:33 dist-test-slave-fg3h krb5kdc[8561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:33.290349 8936 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:30:33.291239 8936 webserver.cc:460] Webserver started at http://127.0.98.3:34383/ using document root <none> and password file <none>
I20260619 12:30:33.291745 8936 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:33.291805 8936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:33.291961 8936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:33.293989 8936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "d7c5517479eb45d0a7b5c6524b398941"
format_stamp: "Formatted at 2026-06-19 12:30:33 on dist-test-slave-fg3h"
server_key: "463db48c920380024d5446f62497511f"
server_key_iv: "4ff588e984dffb063badd8c0555e3ef7"
server_key_version: "encryptionkey@0"
I20260619 12:30:33.294502 8936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d7c5517479eb45d0a7b5c6524b398941"
format_stamp: "Formatted at 2026-06-19 12:30:33 on dist-test-slave-fg3h"
server_key: "463db48c920380024d5446f62497511f"
server_key_iv: "4ff588e984dffb063badd8c0555e3ef7"
server_key_version: "encryptionkey@0"
I20260619 12:30:33.298089 8936 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:30:33.300458 8951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:33.301738 8936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:33.301837 8936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "d7c5517479eb45d0a7b5c6524b398941"
format_stamp: "Formatted at 2026-06-19 12:30:33 on dist-test-slave-fg3h"
server_key: "463db48c920380024d5446f62497511f"
server_key_iv: "4ff588e984dffb063badd8c0555e3ef7"
server_key_version: "encryptionkey@0"
I20260619 12:30:33.301919 8936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:33.326161 8936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:33.329366 8936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:33.329524 8936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:33.330073 8936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:33.330900 8936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:33.330971 8936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:33.331038 8936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:33.331085 8936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:33.340266 8936 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37703
I20260619 12:30:33.340287 9064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37703 every 8 connection(s)
I20260619 12:30:33.341203 8936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:30:33.349504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 8936
I20260619 12:30:33.349664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:30:33.350035 392 external_mini_cluster.cc:1428] Setting key 6c179ea6b829aa28677e6cdc0ebd7b35
Jun 19 12:30:33 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:33.355448 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:33.343097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48829 (local address 127.0.98.62:43419)
0619 12:30:33.343246 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:33.343251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:33.344091 (+ 840us) server_negotiation.cc:407] Connection header received
0619 12:30:33.345651 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:33.345655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:33.345717 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:33.345822 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:33.347026 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.347563 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.348725 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.348952 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.349076 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:33.351823 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:33.351826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:33.351828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:33.351853 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:33.353393 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:33.353937 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:33.353942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:33.353944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:33.354028 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:33.354364 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:33.354370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:33.354371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:33.354569 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:33.354675 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:33.355166 (+ 491us) server_negotiation.cc:299] Negotiation successful
0619 12:30:33.355291 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 12:30:33.356382 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:33.343454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.3:48829)
0619 12:30:33.343925 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:33.343968 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:33.345409 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:33.346000 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:33.346007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:33.346369 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:33.346868 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:33.346881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.347697 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.347700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:33.348594 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:33.348603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.349129 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.349134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:33.349311 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:33.349970 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:33.349993 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:33.351629 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:33.353508 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:33.353515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:33.353530 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:33.353829 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:33.354129 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:33.354132 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:33.354134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:33.354270 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:33.354675 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:33.354683 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:33.354922 (+ 239us) client_negotiation.cc:769] Sending connection context
0619 12:30:33.355106 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 12:30:33.355449 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":126,"threads_started":1}
I20260619 12:30:33.357518 9065 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43419
I20260619 12:30:33.357779 9065 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:33.358291 9065 heartbeater.cc:511] Master 127.0.98.62:43419 requested a full tablet report, sending...
I20260619 12:30:33.359316 8606 ts_manager.cc:194] Registered new tserver with Master: d7c5517479eb45d0a7b5c6524b398941 (127.0.98.3:37703)
I20260619 12:30:33.359881 8606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48829
I20260619 12:30:33.364403 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 12:30:33.439013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 8668
I20260619 12:30:33.444691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 8802
I20260619 12:30:33.449996 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 8936
I20260619 12:30:33.455456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 8574
I20260619 12:30:33.461791 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43419
--webserver_interface=127.0.98.62
--webserver_port=42201
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:33.575168 9074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:33.575453 9074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:33.575533 9074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:33.579595 9074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:33.579679 9074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:33.579715 9074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:33.579766 9074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:33.579802 9074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:30:33.585165 9074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:33.586228 9074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:33.587260 9074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:33.595839 9080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:33.595834 9082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:33.595918 9074 server_base.cc:1034] running on GCE node
W20260619 12:30:33.595832 9079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:33.596608 9074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:33.597692 9074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:33.598846 9074 hybrid_clock.cc:648] HybridClock initialized: now 1781872233598834 us; error 26 us; skew 500 ppm
Jun 19 12:30:33 dist-test-slave-fg3h krb5kdc[8561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:33.601703 9074 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:30:33.602613 9074 webserver.cc:460] Webserver started at http://127.0.98.62:42201/ using document root <none> and password file <none>
I20260619 12:30:33.603155 9074 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:33.603269 9074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:33.606957 9074 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 12:30:33.609026 9089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:33.610303 9074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 12:30:33.610456 9074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "f225e2a39e98498aaab40497179a00ba"
format_stamp: "Formatted at 2026-06-19 12:30:32 on dist-test-slave-fg3h"
server_key: "e8466d2b120e00c13cf90854156c65da"
server_key_iv: "3f7f2c02cce69a847f3f2973e8d8263a"
server_key_version: "encryptionkey@0"
I20260619 12:30:33.610999 9074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:33.631829 9074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:33.634764 9074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:33.634969 9074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:33.642062 9074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43419
I20260619 12:30:33.642099 9141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43419 every 8 connection(s)
I20260619 12:30:33.643113 9074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:30:33.647445 9142 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:30:33.647946 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 9074
I20260619 12:30:33.648440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46235
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43419
--webserver_port=33621
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:30:33.649710 9142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba: Bootstrap starting.
I20260619 12:30:33.666188 9142 log.cc:824] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:33.672685 9142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:30:33.673156 9142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba: Bootstrap complete.
I20260619 12:30:33.677335 9142 raft_consensus.cc:348] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f225e2a39e98498aaab40497179a00ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43419 } }
I20260619 12:30:33.677768 9142 raft_consensus.cc:729] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f225e2a39e98498aaab40497179a00ba, State: Initialized, Role: FOLLOWER
I20260619 12:30:33.678412 9142 consensus_queue.cc:260] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f225e2a39e98498aaab40497179a00ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43419 } }
I20260619 12:30:33.678566 9142 raft_consensus.cc:388] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:30:33.678640 9142 raft_consensus.cc:482] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:30:33.678735 9142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:30:33.680629 9142 raft_consensus.cc:504] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f225e2a39e98498aaab40497179a00ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43419 } }
I20260619 12:30:33.680974 9142 leader_election.cc:302] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f225e2a39e98498aaab40497179a00ba; no voters:
I20260619 12:30:33.681334 9142 leader_election.cc:288] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 12:30:33.681419 9146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [term 2 FOLLOWER]: Leader election won for term 2
I20260619 12:30:33.681653 9146 raft_consensus.cc:686] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [term 2 LEADER]: Becoming Leader. State: Replica: f225e2a39e98498aaab40497179a00ba, State: Running, Role: LEADER
I20260619 12:30:33.681998 9146 consensus_queue.cc:237] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f225e2a39e98498aaab40497179a00ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43419 } }
I20260619 12:30:33.682565 9142 sys_catalog.cc:564] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:33.682937 9146 sys_catalog.cc:455] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader f225e2a39e98498aaab40497179a00ba. Latest consensus state: current_term: 2 leader_uuid: "f225e2a39e98498aaab40497179a00ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f225e2a39e98498aaab40497179a00ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43419 } } }
I20260619 12:30:33.683045 9146 sys_catalog.cc:458] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:33.683665 9147 sys_catalog.cc:455] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f225e2a39e98498aaab40497179a00ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f225e2a39e98498aaab40497179a00ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43419 } } }
I20260619 12:30:33.683761 9147 sys_catalog.cc:458] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:33.684273 9150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:30:33.687426 9150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:30:33.688129 9150 catalog_manager.cc:1259] Loaded cluster ID: f4d7aae1747b4cff80d288b9a33479eb
I20260619 12:30:33.688194 9150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:30:33.690564 9150 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:30:33.691206 9150 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f225e2a39e98498aaab40497179a00ba: Loaded TSK: 0
I20260619 12:30:33.692401 9150 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 12:30:33.761269 9144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:33.761574 9144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:33.761647 9144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:33.765806 9144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:33.765890 9144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:33.765980 9144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:30:33.771179 9144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33621
--tserver_master_addrs=127.0.98.62:43419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:33.772217 9144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:33.773313 9144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:33.782774 9169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:33.782800 9171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:33.782783 9168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:33.783210 9144 server_base.cc:1034] running on GCE node
I20260619 12:30:33.783656 9144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:33.784438 9144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:33.785641 9144 hybrid_clock.cc:648] HybridClock initialized: now 1781872233785620 us; error 35 us; skew 500 ppm
Jun 19 12:30:33 dist-test-slave-fg3h krb5kdc[8561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872233, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:33.792132 9144 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 12:30:33.793315 9144 webserver.cc:460] Webserver started at http://127.0.98.1:33621/ using document root <none> and password file <none>
I20260619 12:30:33.794032 9144 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:33.794144 9144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:33.799317 9144 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 12:30:33.801836 9178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:33.803325 9144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:30:33.803443 9144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "55f3a0c52c884ce89f3ef462e877618d"
format_stamp: "Formatted at 2026-06-19 12:30:32 on dist-test-slave-fg3h"
server_key: "24c8908150e7e59ff1d67fd9f32396f5"
server_key_iv: "05593b1e6e7d24d1f9d4e2e182d585cc"
server_key_version: "encryptionkey@0"
I20260619 12:30:33.804137 9144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:33.828918 9144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:33.832777 9144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:33.832970 9144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:33.833665 9144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:33.834658 9144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:33.834717 9144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:33.834769 9144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:33.834791 9144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:33.844827 9144 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46235
I20260619 12:30:33.844871 9291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46235 every 8 connection(s)
I20260619 12:30:33.845955 9144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:30:33.855338 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 9144
I20260619 12:30:33.855829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33125
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43419
--webserver_port=40173
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:30:33 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:33 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:33.859025 9295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:33.848151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.1:44737)
0619 12:30:33.848616 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:33.848668 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:33.850037 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:33.851116 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:33.851122 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:33.851478 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:33.851937 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:33.851951 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.853236 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.853238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:33.854088 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:33.854097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.854858 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.854861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:33.854994 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:33.855650 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:33.855666 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:33.858271 (+ 2605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":155,"threads_started":1}
W20260619 12:30:33.859524 9292 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43419 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43419: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260619 12:30:33.859977 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:33.848063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44737 (local address 127.0.98.62:43419)
0619 12:30:33.848492 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:33.848509 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:33.848785 (+ 276us) server_negotiation.cc:407] Connection header received
0619 12:30:33.850421 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:33.850433 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:33.850733 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:33.851017 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:33.852097 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.853102 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.854195 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.854687 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.854816 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:33.858590 (+ 3774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44737: BlockingRecv error: recv got EOF from 127.0.98.1:44737 (error 108)
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":129,"threads_started":1}
Jun 19 12:30:33 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:33 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:33.866190 9295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:33.860030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.1:44501)
0619 12:30:33.860363 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:33.860376 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:33.860444 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:33.860817 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:33.860819 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:33.860948 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:33.861034 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:33.861038 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.862192 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.862195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:33.862730 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:33.862737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.863413 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.863416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:33.863524 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:33.864079 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:33.864100 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:33.866015 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260}
W20260619 12:30:33.866226 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:33.860091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44501 (local address 127.0.98.62:43419)
0619 12:30:33.860563 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:33.860570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:33.860588 (+ 18us) server_negotiation.cc:407] Connection header received
0619 12:30:33.860635 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:33.860640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:33.860701 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:33.860797 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:33.861195 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.862086 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.862842 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.863276 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.863403 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:33.866134 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44501: BlockingRecv error: recv got EOF from 127.0.98.1:44501 (error 108)
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":137,"threads_started":1}
Jun 19 12:30:33 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:33 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:33.873759 9295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:33.866886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.1:60323)
0619 12:30:33.866997 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:33.867013 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:33.867086 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:33.867318 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:33.867321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:33.867478 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:33.867541 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:33.867546 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.868269 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.868273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:33.868782 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:33.868790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.869292 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.869296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:33.869374 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:33.870667 (+ 1293us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:33.870688 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:33.873612 (+ 2924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 12:30:33.873872 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:33.866980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60323 (local address 127.0.98.62:43419)
0619 12:30:33.867087 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:33.867091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:33.867106 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:30:33.867167 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:33.867169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:33.867215 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:33.867304 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:33.867649 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.868153 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.868898 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:33.869172 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:33.869286 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:33.873742 (+ 4456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60323: BlockingRecv error: recv got EOF from 127.0.98.1:60323 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 12:30:33.877260 9292 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:30:33.982791 9296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:33.983035 9296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:33.983078 9296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:33.987216 9296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:33.987275 9296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:33.987334 9296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:30:33.992657 9296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40173
--tserver_master_addrs=127.0.98.62:43419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:33.993748 9296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:33.994684 9296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:34.004397 9303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:34.004503 9296 server_base.cc:1034] running on GCE node
W20260619 12:30:34.004410 9302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:34.004415 9305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:34.005182 9296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:34.005775 9296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:34.006944 9296 hybrid_clock.cc:648] HybridClock initialized: now 1781872234006913 us; error 43 us; skew 500 ppm
Jun 19 12:30:34 dist-test-slave-fg3h krb5kdc[8561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872234, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:34.012781 9296 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 12:30:34.013834 9296 webserver.cc:460] Webserver started at http://127.0.98.2:40173/ using document root <none> and password file <none>
I20260619 12:30:34.014442 9296 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:34.014539 9296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:34.018656 9296 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:30:34.020879 9312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:34.022329 9296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:30:34.022483 9296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "f73a4f5dfa85470ba1d20fa4ce0222d0"
format_stamp: "Formatted at 2026-06-19 12:30:33 on dist-test-slave-fg3h"
server_key: "932896148af020025141ccdc3300b37a"
server_key_iv: "db6f71a45392f0881332ce1a80f47025"
server_key_version: "encryptionkey@0"
I20260619 12:30:34.023058 9296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:34.051108 9296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:34.054410 9296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:34.054643 9296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:34.055280 9296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:34.056123 9296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:34.056195 9296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:34.056269 9296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:34.056313 9296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:34.065295 9296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33125
I20260619 12:30:34.065321 9425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33125 every 8 connection(s)
I20260619 12:30:34.066262 9296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:30:34.073395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 9296
I20260619 12:30:34.073824 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37703
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43419
--webserver_port=34383
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:30:34 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:34 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 12:30:34.078348 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:34.068087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41373 (local address 127.0.98.62:43419)
0619 12:30:34.068241 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:34.068247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:34.069110 (+ 863us) server_negotiation.cc:407] Connection header received
0619 12:30:34.070510 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:34.070513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:34.070560 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:34.070681 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:34.071850 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.072401 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.073783 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.074104 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.074231 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:34.078157 (+ 3926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41373: BlockingRecv error: recv got EOF from 127.0.98.2:41373 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 12:30:34.078823 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:34.068524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.2:41373)
0619 12:30:34.068956 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:34.068997 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:34.070309 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:34.070836 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:34.070843 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:34.071181 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:34.071700 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:34.071714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.072560 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.072598 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:34.073644 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:34.073655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.074271 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.074275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:34.074452 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:34.075135 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:34.075162 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:34.077948 (+ 2786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":113,"threads_started":1}
W20260619 12:30:34.079447 9426 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43419 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43419: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 12:30:34 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:34 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:34.085949 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:34.079940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.2:44939)
0619 12:30:34.080056 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:34.080090 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:34.080172 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:34.080446 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:34.080447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:34.080578 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:34.080675 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:34.080680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.081841 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.081844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:34.082412 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:34.082422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.083095 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.083098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:34.083167 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:34.083730 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:34.083742 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:34.085851 (+ 2109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 12:30:34.086119 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:34.080030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44939 (local address 127.0.98.62:43419)
0619 12:30:34.080184 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:34.080190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:34.080208 (+ 18us) server_negotiation.cc:407] Connection header received
0619 12:30:34.080263 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:34.080268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:34.080329 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:34.080429 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:34.080793 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.081719 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.082530 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.082947 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.083062 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:34.086009 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44939: BlockingRecv error: recv got EOF from 127.0.98.2:44939 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 12:30:34 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:34 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:34.091974 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:34.086652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.2:38017)
0619 12:30:34.086836 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:34.086848 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:34.086907 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:34.087129 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:34.087131 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:34.087248 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:34.087316 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:34.087320 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.088346 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.088349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:34.088911 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:34.088924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.089492 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.089494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:34.089593 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:34.090081 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:34.090092 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:34.091884 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113}
W20260619 12:30:34.092163 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:34.086755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38017 (local address 127.0.98.62:43419)
0619 12:30:34.086867 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:34.086871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:34.086884 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:30:34.086983 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:34.086986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:34.087026 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:34.087106 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:34.087405 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.088212 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.089026 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.089391 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.089488 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:34.092053 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38017: BlockingRecv error: recv got EOF from 127.0.98.2:38017 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 12:30:34.092303 9426 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:30:34.190174 9429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:34.190459 9429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:34.190531 9429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:34.194655 9429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:34.194721 9429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:34.194815 9429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:30:34.200165 9429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34383
--tserver_master_addrs=127.0.98.62:43419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:34.201341 9429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:34.202373 9429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:30:34.212164 9429 server_base.cc:1034] running on GCE node
W20260619 12:30:34.212100 9434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:34.212132 9435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:34.212110 9437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:34.212790 9429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:34.213470 9429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:34.214641 9429 hybrid_clock.cc:648] HybridClock initialized: now 1781872234214615 us; error 30 us; skew 500 ppm
Jun 19 12:30:34 dist-test-slave-fg3h krb5kdc[8561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872234, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:34.217451 9429 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 12:30:34.218341 9429 webserver.cc:460] Webserver started at http://127.0.98.3:34383/ using document root <none> and password file <none>
I20260619 12:30:34.218885 9429 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:34.218954 9429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:34.222663 9429 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:34.224880 9444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:34.226154 9429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:30:34.226300 9429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "d7c5517479eb45d0a7b5c6524b398941"
format_stamp: "Formatted at 2026-06-19 12:30:33 on dist-test-slave-fg3h"
server_key: "463db48c920380024d5446f62497511f"
server_key_iv: "4ff588e984dffb063badd8c0555e3ef7"
server_key_version: "encryptionkey@0"
I20260619 12:30:34.226871 9429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:34.246146 9429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:34.249464 9429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:34.249655 9429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:34.250232 9429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:34.251022 9429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:34.251067 9429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:34.251130 9429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:34.251165 9429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:34.260242 9429 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37703
I20260619 12:30:34.260268 9557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37703 every 8 connection(s)
I20260619 12:30:34.261294 9429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 12:30:34 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:34.270908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 9429
Jun 19 12:30:34 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 12:30:34.272573 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:34.263077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47661 (local address 127.0.98.62:43419)
0619 12:30:34.263208 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:34.263212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:34.264033 (+ 821us) server_negotiation.cc:407] Connection header received
0619 12:30:34.265439 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:34.265442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:34.265496 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:34.265576 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:34.266748 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.267278 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.268410 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.268646 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.268762 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:34.272424 (+ 3662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47661: BlockingRecv error: recv got EOF from 127.0.98.3:47661 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 12:30:34.273319 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:34.263453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.3:47661)
0619 12:30:34.263889 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:34.263931 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:34.265225 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:34.265747 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:34.265755 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:34.266093 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:34.266598 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:34.266611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.267413 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.267417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:34.268272 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:34.268290 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.268801 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.268806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:34.268977 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:34.269643 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:34.269659 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:34.272290 (+ 2631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":121,"threads_started":1}
W20260619 12:30:34.273821 9558 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43419 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43419: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 12:30:34 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:34 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:34.280074 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:34.274336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.3:34889)
0619 12:30:34.274544 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:34.274558 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:34.274642 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:34.274904 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:34.274907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:34.275084 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:34.275188 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:34.275194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.276121 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.276125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:34.276666 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:34.276674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.277210 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.277213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:34.277328 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:34.277892 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:34.277903 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:34.279928 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137}
W20260619 12:30:34.280138 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:34.274406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34889 (local address 127.0.98.62:43419)
0619 12:30:34.274520 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:34.274524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:34.274601 (+ 77us) server_negotiation.cc:407] Connection header received
0619 12:30:34.274723 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:34.274726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:34.274771 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:34.274887 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:34.275307 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.275982 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.276787 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.277064 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.277207 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:34.280031 (+ 2824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34889: BlockingRecv error: recv got EOF from 127.0.98.3:34889 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 12:30:34.281742 9297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:34.271999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.0.98.62:43419)
0619 12:30:34.272150 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:34.272167 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:34.272183 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:30:34.272228 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:34.272232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:34.272283 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:34.272373 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:34.273292 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.274066 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.275000 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.275346 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.275429 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:34.276955 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:34.276961 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:34.276976 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:34.277001 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:34.279286 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:34.279849 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:34.279855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:34.279860 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:34.279983 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:34.280450 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:34.280452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:34.280453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:34.280894 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:34.281134 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:34.281403 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 12:30:34.281643 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 12:30:34 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:34 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:34.285905 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:34.280672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.3:42115)
0619 12:30:34.280774 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:34.280790 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:34.280864 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:34.281078 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:34.281081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:34.281257 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:34.281328 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:34.281332 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.282272 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.282277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:34.283030 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:34.283041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.283515 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.283518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:34.283577 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:34.284074 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:34.284085 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:34.285818 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 12:30:34.286036 9294 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:34.280753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42115 (local address 127.0.98.62:43419)
0619 12:30:34.280881 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:34.280886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:34.280902 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:30:34.280937 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:34.280940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:34.280982 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:34.281065 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:34.281436 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.282150 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.283151 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.283369 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.283479 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:34.285935 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42115: BlockingRecv error: recv got EOF from 127.0.98.3:42115 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 12:30:34.286283 9558 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 12:30:34 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:34 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:34.884037 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:34.878029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.1:45775)
0619 12:30:34.878467 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:34.878484 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:34.878609 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:34.878842 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:34.878845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:34.879017 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:34.879122 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:34.879127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.879908 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.879910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:34.880521 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:34.880529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.881079 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.881110 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:34.881210 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:34.881795 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:34.881815 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:34.883862 (+ 2047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":91,"threads_started":1}
W20260619 12:30:34.884153 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:34.878112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45775 (local address 127.0.98.62:43419)
0619 12:30:34.878456 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:34.878461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:34.878550 (+ 89us) server_negotiation.cc:407] Connection header received
0619 12:30:34.878679 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:34.878681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:34.878728 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:34.878807 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:34.879227 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.879804 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.880675 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:34.880927 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:34.881068 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:34.883988 (+ 2920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45775: BlockingRecv error: recv got EOF from 127.0.98.1:45775 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":89,"threads_started":1}
Jun 19 12:30:35 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:35 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:35.099411 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:35.093067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.2:55523)
0619 12:30:35.093419 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:35.093432 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:35.093571 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:35.093873 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:35.093879 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:35.094111 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:35.094265 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:35.094271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:35.095153 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:35.095156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:35.095722 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:35.095730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:35.096175 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:35.096178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:35.096255 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:35.096807 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:35.096818 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:35.099210 (+ 2392us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":95,"threads_started":1}
W20260619 12:30:35.099468 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:35.093210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55523 (local address 127.0.98.62:43419)
0619 12:30:35.093370 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:35.093375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:35.093538 (+ 163us) server_negotiation.cc:407] Connection header received
0619 12:30:35.093682 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:35.093692 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:35.093743 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:35.093819 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:35.094426 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:35.095040 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:35.095847 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:35.096070 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:35.096161 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:35.099347 (+ 3186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55523: BlockingRecv error: recv got EOF from 127.0.98.2:55523 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 12:30:35 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:35 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:35.293126 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:35.286923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.3:52153)
0619 12:30:35.287325 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:35.287340 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:35.287485 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:35.287807 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:35.287810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:35.288015 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:35.288120 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:35.288127 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:35.288887 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:35.288891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:35.289740 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:35.289763 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:35.290418 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:35.290421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:35.290534 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:35.291078 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:35.291089 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:35.292926 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":89,"threads_started":1}
W20260619 12:30:35.293188 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:35.286985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52153 (local address 127.0.98.62:43419)
0619 12:30:35.287294 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:35.287299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:35.287430 (+ 131us) server_negotiation.cc:407] Connection header received
0619 12:30:35.287597 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:35.287601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:35.287655 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:35.287745 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:35.288242 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:35.288773 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:35.289916 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:35.290272 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:35.290401 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:35.293042 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52153: BlockingRecv error: recv got EOF from 127.0.98.3:52153 (error 108)
Metrics: {"server-negotiator.queue_time_us":200}
Jun 19 12:30:35 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:35 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:35.890338 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:35.885064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.1:60281)
0619 12:30:35.885349 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:35.885370 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:35.885449 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:35.885640 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:35.885642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:35.885816 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:35.885908 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:35.885913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:35.886642 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:35.886645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:35.887224 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:35.887233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:35.887693 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:35.887695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:35.887760 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:35.888244 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:35.888255 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:35.890140 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":87,"threads_started":1}
W20260619 12:30:35.890377 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:35.885145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60281 (local address 127.0.98.62:43419)
0619 12:30:35.885392 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:35.885396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:35.885425 (+ 29us) server_negotiation.cc:407] Connection header received
0619 12:30:35.885520 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:35.885522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:35.885563 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:35.885630 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:35.886003 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:35.886537 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:35.887353 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:35.887593 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:35.887680 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:35.890260 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60281: BlockingRecv error: recv got EOF from 127.0.98.1:60281 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":103,"threads_started":1}
Jun 19 12:30:36 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:36 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:36.105885 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:36.100320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.2:48807)
0619 12:30:36.100598 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:36.100611 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:36.100709 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:36.100944 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:36.100946 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:36.101155 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:36.101252 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:36.101266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:36.102056 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:36.102059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:36.102643 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:36.102651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:36.103127 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:36.103130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:36.103190 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:36.103753 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:36.103767 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:36.105721 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":109,"threads_started":1}
W20260619 12:30:36.106032 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:36.100370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48807 (local address 127.0.98.62:43419)
0619 12:30:36.100526 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:36.100529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:36.100681 (+ 152us) server_negotiation.cc:407] Connection header received
0619 12:30:36.100787 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:36.100789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:36.100832 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:36.100905 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:36.101415 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:36.101943 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:36.102796 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:36.103019 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:36.103099 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:36.105911 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48807: BlockingRecv error: recv got EOF from 127.0.98.2:48807 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 12:30:36 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:36 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:36.299851 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:36.294018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.3:38475)
0619 12:30:36.294327 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:36.294343 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:36.294448 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:36.294738 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:36.294742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:36.294936 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:36.295087 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:36.295097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:36.296020 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:36.296023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:36.296588 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:36.296597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:36.297044 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:36.297046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:36.297185 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:36.297758 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:36.297770 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:36.299661 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":101,"threads_started":1}
W20260619 12:30:36.299898 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:36.294203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38475 (local address 127.0.98.62:43419)
0619 12:30:36.294373 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:36.294376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:36.294390 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:30:36.294541 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:36.294546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:36.294599 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:36.294676 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:36.295212 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:36.295910 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:36.296709 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:36.296939 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:36.297023 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:36.299794 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38475: BlockingRecv error: recv got EOF from 127.0.98.3:38475 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 12:30:36 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:36 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:36.896510 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:36.891175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.1:36485)
0619 12:30:36.891445 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:36.891459 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:36.891546 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:36.891799 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:36.891801 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:36.891983 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:36.892083 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:36.892088 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:36.892858 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:36.892860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:36.893463 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:36.893472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:36.893965 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:36.893968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:36.894035 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:36.894541 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:36.894553 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:36.896397 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
W20260619 12:30:36.896729 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:36.891238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36485 (local address 127.0.98.62:43419)
0619 12:30:36.891532 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:36.891536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:36.891550 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:30:36.891631 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:36.891633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:36.891677 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:36.891803 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:36.892215 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:36.892756 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:36.893582 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:36.893854 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:36.893940 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:36.896585 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36485: BlockingRecv error: recv got EOF from 127.0.98.1:36485 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":126,"threads_started":1}
Jun 19 12:30:37 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:37 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 12:30:37.112807 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:37.106796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52299 (local address 127.0.98.62:43419)
0619 12:30:37.107014 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:37.107019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:37.107184 (+ 165us) server_negotiation.cc:407] Connection header received
0619 12:30:37.107373 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:37.107377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:37.107425 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:37.107521 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:37.107970 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:37.108564 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:37.109510 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:37.109743 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:37.109858 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:37.112709 (+ 2851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52299: BlockingRecv error: recv got EOF from 127.0.98.2:52299 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 12:30:37.112771 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:37.106646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.2:52299)
0619 12:30:37.107092 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:37.107105 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:37.107250 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:37.107540 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:37.107543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:37.107756 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:37.107845 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:37.107851 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:37.108695 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:37.108698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:37.109382 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:37.109391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:37.109870 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:37.109873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:37.109958 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:37.110508 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:37.110520 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:37.112568 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":151,"threads_started":1}
Jun 19 12:30:37 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:37 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:37.307246 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:37.300739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.3:33511)
0619 12:30:37.301122 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:37.301136 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:37.301282 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:37.301603 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:37.301606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:37.301840 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:37.301946 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:37.301952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:37.302801 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:37.302807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:37.303466 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:37.303475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:37.304010 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:37.304014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:37.304111 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:37.304692 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:37.304705 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:37.307119 (+ 2414us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":95,"threads_started":1}
W20260619 12:30:37.307495 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:37.300876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33511 (local address 127.0.98.62:43419)
0619 12:30:37.301051 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:37.301056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:37.301220 (+ 164us) server_negotiation.cc:407] Connection header received
0619 12:30:37.301414 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:37.301418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:37.301475 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:37.301573 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:37.302084 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:37.302668 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:37.303607 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:37.303871 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:37.303997 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:37.307353 (+ 3356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33511: BlockingRecv error: recv got EOF from 127.0.98.3:33511 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 12:30:37 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:37 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:37.903913 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:37.897461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.1:33637)
0619 12:30:37.897808 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:37.897829 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:37.897988 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:37.898269 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:37.898271 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:37.898486 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:37.898572 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:37.898577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:37.899374 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:37.899378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:37.900039 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:37.900048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:37.900521 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:37.900525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:37.900609 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:37.901275 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:37.901294 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:37.903679 (+ 2385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":95,"threads_started":1}
W20260619 12:30:37.904564 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:37.897484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33637 (local address 127.0.98.62:43419)
0619 12:30:37.897745 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:37.897749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:37.897932 (+ 183us) server_negotiation.cc:407] Connection header received
0619 12:30:37.898098 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:37.898101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:37.898158 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:37.898253 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:37.898686 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:37.899261 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:37.900170 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:37.900411 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:37.900519 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:37.904390 (+ 3871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33637: BlockingRecv error: recv got EOF from 127.0.98.1:33637 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
Jun 19 12:30:38 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:38 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:38.119071 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:38.113731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.2:39671)
0619 12:30:38.113986 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:38.114003 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:38.114108 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:38.114339 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:38.114341 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:38.114538 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:38.114649 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:38.114655 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:38.115420 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:38.115423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:38.115989 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:38.115997 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:38.116410 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:38.116413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:38.116480 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:38.116989 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:38.117001 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:38.118946 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":89,"threads_started":1}
W20260619 12:30:38.119232 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:38.113796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39671 (local address 127.0.98.62:43419)
0619 12:30:38.113905 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:38.113934 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:38.114057 (+ 123us) server_negotiation.cc:407] Connection header received
0619 12:30:38.114173 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:38.114175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:38.114218 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:38.114289 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:38.114746 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:38.115312 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:38.116093 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:38.116298 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:38.116373 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:38.119093 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39671: BlockingRecv error: recv got EOF from 127.0.98.2:39671 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 19 12:30:38 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:38 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:38.314531 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:38.308137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.3:40653)
0619 12:30:38.308424 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:38.308438 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:38.308537 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:38.308888 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:38.308891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:38.309178 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:38.309282 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:38.309288 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:38.310307 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:38.310313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:38.311115 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:38.311126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:38.311631 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:38.311635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:38.311713 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:38.312247 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:38.312264 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:38.314403 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
W20260619 12:30:38.314684 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:38.308282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40653 (local address 127.0.98.62:43419)
0619 12:30:38.308467 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:38.308471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:38.308490 (+ 19us) server_negotiation.cc:407] Connection header received
0619 12:30:38.308653 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:38.308658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:38.308747 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:38.308858 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:38.309421 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:38.310184 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:38.311275 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:38.311500 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:38.311589 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:38.314567 (+ 2978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40653: BlockingRecv error: recv got EOF from 127.0.98.3:40653 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 19 12:30:38 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:38 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:38.916035 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:38.910282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.1:45143)
0619 12:30:38.910576 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:38.910591 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:38.910686 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:38.910969 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:38.910972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:38.911162 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:38.911255 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:38.911261 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:38.912089 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:38.912092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:38.912679 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:38.912687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:38.913240 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:38.913243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:38.913317 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:38.913862 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:38.913875 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:38.915906 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":73,"threads_started":1}
W20260619 12:30:38.916224 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:38.910362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45143 (local address 127.0.98.62:43419)
0619 12:30:38.910599 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:38.910602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:38.910645 (+ 43us) server_negotiation.cc:407] Connection header received
0619 12:30:38.910771 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:38.910774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:38.910827 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:38.910903 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:38.911389 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:38.911979 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:38.912800 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:38.913029 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:38.913203 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:38.916046 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45143: BlockingRecv error: recv got EOF from 127.0.98.1:45143 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":102,"threads_started":1}
Jun 19 12:30:39 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:39 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:39.129606 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:39.119832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.2:53085)
0619 12:30:39.120154 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:39.120171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:39.120277 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:39.120561 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:39.120563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:39.120766 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:39.120855 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:39.120860 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:39.121703 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:39.121708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:39.122281 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:39.122288 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:39.122747 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:39.122751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:39.122836 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:39.127285 (+ 4449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:39.127299 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:39.129470 (+ 2171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":102,"threads_started":1}
W20260619 12:30:39.129784 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:39.120040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53085 (local address 127.0.98.62:43419)
0619 12:30:39.120217 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:39.120223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:39.120238 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:30:39.120387 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:39.120392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:39.120446 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:39.120540 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:39.120984 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:39.121590 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:39.122401 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:39.122633 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:39.122742 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:39.129642 (+ 6900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53085: BlockingRecv error: recv got EOF from 127.0.98.2:53085 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 19 12:30:39 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:39 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:39.321563 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:39.315446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.3:49701)
0619 12:30:39.315780 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:39.315799 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:39.315898 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:39.316142 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:39.316144 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:39.316323 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:39.316442 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:39.316448 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:39.317278 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:39.317281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:39.317916 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:39.317925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:39.318425 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:39.318428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:39.318496 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:39.319092 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:39.319105 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:39.321450 (+ 2345us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":161,"threads_started":1}
W20260619 12:30:39.321686 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:39.315595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49701 (local address 127.0.98.62:43419)
0619 12:30:39.315743 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:39.315747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:39.315899 (+ 152us) server_negotiation.cc:407] Connection header received
0619 12:30:39.315974 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:39.315977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:39.316025 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:39.316103 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:39.316564 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:39.317154 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:39.318060 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:39.318283 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:39.318375 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:39.321571 (+ 3196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49701: BlockingRecv error: recv got EOF from 127.0.98.3:49701 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 12:30:39 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:39 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:39.922587 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:39.916830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.1:35943)
0619 12:30:39.917289 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:39.917305 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:39.917415 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:39.917660 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:39.917663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:39.917844 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:39.917974 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:39.917980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:39.918777 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:39.918780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:39.919339 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:39.919347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:39.919779 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:39.919782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:39.919842 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:39.920491 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:39.920503 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:39.922473 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":89,"threads_started":1}
W20260619 12:30:39.922827 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:39.916880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35943 (local address 127.0.98.62:43419)
0619 12:30:39.917212 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:39.917216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:39.917379 (+ 163us) server_negotiation.cc:407] Connection header received
0619 12:30:39.917491 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:39.917494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:39.917541 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:39.917620 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:39.918088 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:39.918658 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:39.919445 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:39.919673 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:39.919754 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:39.922660 (+ 2906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35943: BlockingRecv error: recv got EOF from 127.0.98.1:35943 (error 108)
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":89,"threads_started":1}
Jun 19 12:30:40 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:40 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:40.136204 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:40.130734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.2:52577)
0619 12:30:40.131006 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:40.131023 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:40.131200 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:40.131482 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:40.131484 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:40.131699 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:40.131789 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:40.131794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:40.132550 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:40.132556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:40.133162 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:40.133170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:40.133630 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:40.133633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:40.133712 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:40.134251 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:40.134263 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:40.136007 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":82,"threads_started":1}
W20260619 12:30:40.136227 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:40.130805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52577 (local address 127.0.98.62:43419)
0619 12:30:40.130984 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:40.130989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:40.131142 (+ 153us) server_negotiation.cc:407] Connection header received
0619 12:30:40.131314 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:40.131318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:40.131372 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:40.131462 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:40.131911 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:40.132436 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:40.133281 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:40.133507 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:40.133625 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:40.136131 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52577: BlockingRecv error: recv got EOF from 127.0.98.2:52577 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 12:30:40 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:40 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:40.328339 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:40.322367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.3:55705)
0619 12:30:40.322655 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:40.322669 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:40.322766 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:40.323031 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:40.323033 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:40.323221 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:40.323316 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:40.323321 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:40.324076 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:40.324079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:40.324629 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:40.324637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:40.325249 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:40.325253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:40.325319 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:40.325879 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:40.325891 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:40.328147 (+ 2256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":122,"threads_started":1}
W20260619 12:30:40.328430 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:40.322469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55705 (local address 127.0.98.62:43419)
0619 12:30:40.322606 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:40.322609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:40.322763 (+ 154us) server_negotiation.cc:407] Connection header received
0619 12:30:40.322833 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:40.322835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:40.322883 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:40.322955 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:40.323441 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:40.323969 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:40.324764 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:40.325058 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:40.325246 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:40.328296 (+ 3050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55705: BlockingRecv error: recv got EOF from 127.0.98.3:55705 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 12:30:40 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:40 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:40.929229 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:40.923448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.1:56285)
0619 12:30:40.923752 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:40.923765 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:40.923868 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:40.924113 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:40.924115 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:40.924291 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:40.924381 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:40.924385 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:40.925225 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:40.925227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:40.925781 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:40.925788 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:40.926225 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:40.926228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:40.926294 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:40.926931 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:40.926943 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:40.929109 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":85,"threads_started":1}
W20260619 12:30:40.929463 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:40.923512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56285 (local address 127.0.98.62:43419)
0619 12:30:40.923763 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:40.923767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:40.923838 (+ 71us) server_negotiation.cc:407] Connection header received
0619 12:30:40.923951 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:40.923954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:40.923998 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:40.924079 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:40.924490 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:40.925045 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:40.925878 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:40.926098 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:40.926176 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:40.929303 (+ 3127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56285: BlockingRecv error: recv got EOF from 127.0.98.1:56285 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
Jun 19 12:30:41 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:41 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:41.142838 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:41.137073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.2:50953)
0619 12:30:41.137397 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:41.137410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:41.137543 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:41.137799 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:41.137802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:41.137983 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:41.138122 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:41.138129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:41.138843 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:41.138845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:41.139422 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:41.139429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:41.139974 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:41.139981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:41.140095 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:41.140589 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:41.140601 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:41.142669 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":99,"threads_started":1}
W20260619 12:30:41.142901 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:41.137158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50953 (local address 127.0.98.62:43419)
0619 12:30:41.137312 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:41.137316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:41.137489 (+ 173us) server_negotiation.cc:407] Connection header received
0619 12:30:41.137650 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:41.137652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:41.137700 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:41.137766 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:41.138235 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:41.138751 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:41.139547 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:41.139812 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:41.140017 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:41.142793 (+ 2776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50953: BlockingRecv error: recv got EOF from 127.0.98.2:50953 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 12:30:41 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:41 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:41.335054 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:41.329238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.3:57493)
0619 12:30:41.329558 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:41.329570 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:41.329702 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:41.329981 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:41.329983 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:41.330201 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:41.330294 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:41.330300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:41.331115 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:41.331119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:41.331688 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:41.331697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:41.332168 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:41.332171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:41.332259 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:41.332772 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:41.332783 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:41.334868 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":76,"threads_started":1}
W20260619 12:30:41.335093 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:41.329289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57493 (local address 127.0.98.62:43419)
0619 12:30:41.329430 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:41.329435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:41.329666 (+ 231us) server_negotiation.cc:407] Connection header received
0619 12:30:41.329809 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:41.329812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:41.329865 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:41.329961 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:41.330416 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:41.330970 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:41.331812 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:41.332047 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:41.332165 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:41.334993 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57493: BlockingRecv error: recv got EOF from 127.0.98.3:57493 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 12:30:41 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:41 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:41.936458 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:41.930245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.1:57171)
0619 12:30:41.930599 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:41.930618 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:41.930756 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:41.931076 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:41.931080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:41.931319 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:41.931428 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:41.931433 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:41.932403 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:41.932406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:41.933027 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:41.933036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:41.933553 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:41.933556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:41.933618 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:41.934215 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:41.934227 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:41.936342 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":91,"threads_started":1}
W20260619 12:30:41.936642 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:41.930461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57171 (local address 127.0.98.62:43419)
0619 12:30:41.930727 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:41.930739 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:41.930754 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:30:41.930876 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:41.930879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:41.930933 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:41.931030 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:41.931556 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:41.932254 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:41.933200 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:41.933447 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:41.933534 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:41.936470 (+ 2936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57171: BlockingRecv error: recv got EOF from 127.0.98.1:57171 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
Jun 19 12:30:42 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:42 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:42.149291 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:42.143689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.2:58895)
0619 12:30:42.143931 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:42.143953 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:42.144088 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:42.144348 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:42.144350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:42.144524 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:42.144620 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:42.144624 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:42.145502 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:42.145504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:42.146070 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:42.146077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:42.146513 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:42.146516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:42.146587 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:42.147118 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:42.147134 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:42.149116 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":87,"threads_started":1}
W20260619 12:30:42.149340 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:42.143759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58895 (local address 127.0.98.62:43419)
0619 12:30:42.143897 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:42.143901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:42.144024 (+ 123us) server_negotiation.cc:407] Connection header received
0619 12:30:42.144173 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:42.144175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:42.144218 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:42.144294 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:42.144718 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:42.145403 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:42.146169 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:42.146386 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:42.146461 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:42.149239 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58895: BlockingRecv error: recv got EOF from 127.0.98.2:58895 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 12:30:42 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:42 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:42.341885 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:42.336046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.3:39799)
0619 12:30:42.336383 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:42.336397 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:42.336502 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:42.336755 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:42.336757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:42.336933 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:42.337028 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:42.337034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:42.337903 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:42.337908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:42.338488 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:42.338496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:42.338985 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:42.338989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:42.339095 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:42.339675 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:42.339687 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:42.341711 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":68,"threads_started":1}
W20260619 12:30:42.341954 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:42.336272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39799 (local address 127.0.98.62:43419)
0619 12:30:42.336407 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:42.336411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:42.336453 (+ 42us) server_negotiation.cc:407] Connection header received
0619 12:30:42.336597 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:42.336599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:42.336646 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:42.336715 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:42.337250 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:42.337781 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:42.338631 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:42.338858 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:42.338972 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:42.341843 (+ 2871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39799: BlockingRecv error: recv got EOF from 127.0.98.3:39799 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 12:30:42 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:42 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:42.943158 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:42.937319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.1:38677)
0619 12:30:42.937596 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:42.937609 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:42.937710 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:42.937958 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:42.937960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:42.938136 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:42.938225 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:42.938231 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:42.938997 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:42.939000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:42.939592 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:42.939601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:42.940063 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:42.940066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:42.940164 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:42.940850 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:42.940865 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:42.943034 (+ 2169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":103,"threads_started":1}
W20260619 12:30:42.943344 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:42.937437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38677 (local address 127.0.98.62:43419)
0619 12:30:42.937660 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:42.937663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:42.937677 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:30:42.937796 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:42.937799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:42.937841 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:42.937925 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:42.938329 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:42.938859 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:42.939713 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:42.939933 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:42.940015 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:42.943168 (+ 3153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38677: BlockingRecv error: recv got EOF from 127.0.98.1:38677 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":86,"threads_started":1}
Jun 19 12:30:43 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:43 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:43.156122 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:43.150139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.2:53205)
0619 12:30:43.150423 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:43.150437 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:43.150538 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:43.150783 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:43.150785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:43.150965 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:43.151092 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:43.151098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:43.151878 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:43.151881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:43.152463 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:43.152472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:43.153333 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:43.153337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:43.153430 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:43.154034 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:43.154046 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:43.155991 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":106,"threads_started":1}
W20260619 12:30:43.156337 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:43.150250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53205 (local address 127.0.98.62:43419)
0619 12:30:43.150390 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:43.150394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:43.150506 (+ 112us) server_negotiation.cc:407] Connection header received
0619 12:30:43.150624 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:43.150626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:43.150670 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:43.150746 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:43.151228 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:43.151757 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:43.152728 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:43.152947 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:43.153030 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:43.156200 (+ 3170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53205: BlockingRecv error: recv got EOF from 127.0.98.2:53205 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 12:30:43 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:43 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:43.348556 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:43.342838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.3:43101)
0619 12:30:43.343174 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:43.343187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:43.343290 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:43.343601 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:43.343604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:43.343802 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:43.343903 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:43.343909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:43.344683 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:43.344685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:43.345424 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:43.345434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:43.345848 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:43.345851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:43.345911 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:43.346455 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:43.346465 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:43.348372 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":103,"threads_started":1}
W20260619 12:30:43.348594 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:43.342866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43101 (local address 127.0.98.62:43419)
0619 12:30:43.343030 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:43.343035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:43.343252 (+ 217us) server_negotiation.cc:407] Connection header received
0619 12:30:43.343374 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:43.343377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:43.343431 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:43.343519 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:43.344022 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:43.344578 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:43.345532 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:43.345750 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:43.345830 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:43.348493 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43101: BlockingRecv error: recv got EOF from 127.0.98.3:43101 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 12:30:43 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:43 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:43.951333 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:43.943980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.1:36597)
0619 12:30:43.944262 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:43.944274 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:43.944355 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:43.944683 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:43.944686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:43.944907 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:43.945049 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:43.945056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:43.946173 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:43.946178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:43.947135 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:43.947147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:43.947700 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:43.947707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:43.947827 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:43.948491 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:43.948506 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:43.951063 (+ 2557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":64,"threads_started":1}
W20260619 12:30:43.951757 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:43.944054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36597 (local address 127.0.98.62:43419)
0619 12:30:43.944305 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:43.944312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:43.944330 (+ 18us) server_negotiation.cc:407] Connection header received
0619 12:30:43.944462 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:43.944465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:43.944521 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:43.944626 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:43.945200 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:43.946028 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:43.947264 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:43.947564 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:43.947674 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:43.951232 (+ 3558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36597: BlockingRecv error: recv got EOF from 127.0.98.1:36597 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":78,"threads_started":1}
Jun 19 12:30:44 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:44 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:44.167441 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:44.156938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.2:53115)
0619 12:30:44.157348 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:44.157361 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:44.157508 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:44.157796 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:44.157799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:44.158011 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:44.158106 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:44.158112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:44.158905 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:44.158910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:44.159476 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:44.159483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:44.159951 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:44.159954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:44.160040 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:44.165232 (+ 5192us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:44.165247 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:44.167320 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":106,"threads_started":1}
W20260619 12:30:44.167661 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:44.157068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53115 (local address 127.0.98.62:43419)
0619 12:30:44.157268 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:44.157273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:44.157444 (+ 171us) server_negotiation.cc:407] Connection header received
0619 12:30:44.157612 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:44.157616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:44.157669 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:44.157774 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:44.158247 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:44.158787 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:44.159599 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:44.159830 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:44.159941 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:44.167510 (+ 7569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53115: BlockingRecv error: recv got EOF from 127.0.98.2:53115 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 19 12:30:44 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:44 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:44.355417 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:44.349435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.3:34211)
0619 12:30:44.349855 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:44.349875 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:44.350027 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:44.350377 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:44.350380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:44.350620 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:44.350739 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:44.350745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:44.351514 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:44.351519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:44.352104 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:44.352113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:44.352586 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:44.352590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:44.352685 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:44.353277 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:44.353298 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:44.355244 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":97,"threads_started":1}
W20260619 12:30:44.355506 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:44.349636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34211 (local address 127.0.98.62:43419)
0619 12:30:44.349766 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:44.349769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:44.349972 (+ 203us) server_negotiation.cc:407] Connection header received
0619 12:30:44.350129 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:44.350134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:44.350238 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:44.350325 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:44.350864 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:44.351396 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:44.352246 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:44.352472 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:44.352584 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:44.355383 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34211: BlockingRecv error: recv got EOF from 127.0.98.3:34211 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 12:30:44 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:44 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:44.958025 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:44.952171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.1:58137)
0619 12:30:44.952447 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:44.952461 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:44.952547 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:44.952890 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:44.952891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:44.953071 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:44.953195 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:44.953201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:44.954011 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:44.954013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:44.954584 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:44.954592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:44.955028 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:44.955030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:44.955088 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:44.955673 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:44.955685 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:44.957854 (+ 2169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":102,"threads_started":1}
W20260619 12:30:44.958143 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:44.952270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58137 (local address 127.0.98.62:43419)
0619 12:30:44.952535 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:44.952539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:44.952552 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:30:44.952712 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:44.952715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:44.952767 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:44.952856 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:44.953345 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:44.953908 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:44.954703 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:44.954926 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:44.955009 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:44.957996 (+ 2987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58137: BlockingRecv error: recv got EOF from 127.0.98.1:58137 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":125,"threads_started":1}
Jun 19 12:30:45 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:45 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:45.174434 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:45.168399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.2:43859)
0619 12:30:45.168678 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:45.168692 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:45.168797 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:45.169199 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:45.169201 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:45.169402 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:45.169501 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:45.169507 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:45.170315 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:45.170320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:45.170923 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:45.170931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:45.171433 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:45.171437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:45.171522 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:45.172129 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:45.172141 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:45.174246 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":95,"threads_started":1}
W20260619 12:30:45.174507 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:45.168474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43859 (local address 127.0.98.62:43419)
0619 12:30:45.168629 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:45.168636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:45.168769 (+ 133us) server_negotiation.cc:407] Connection header received
0619 12:30:45.168921 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:45.168924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:45.168972 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:45.169064 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:45.169617 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:45.170129 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:45.171051 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:45.171293 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:45.171410 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:45.174385 (+ 2975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43859: BlockingRecv error: recv got EOF from 127.0.98.2:43859 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 12:30:45 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:45 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:45.361974 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:45.356221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.3:45213)
0619 12:30:45.356516 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:45.356530 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:45.356628 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:45.356911 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:45.356913 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:45.357125 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:45.357313 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:45.357320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:45.358230 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:45.358236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:45.358964 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:45.358973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:45.359381 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:45.359384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:45.359442 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:45.359940 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:45.359952 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:45.361858 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":125,"threads_started":1}
W20260619 12:30:45.362118 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:45.356370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45213 (local address 127.0.98.62:43419)
0619 12:30:45.356510 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:45.356513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:45.356601 (+ 88us) server_negotiation.cc:407] Connection header received
0619 12:30:45.356707 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:45.356709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:45.356755 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:45.356836 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:45.357478 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:45.358117 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:45.359066 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:45.359281 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:45.359358 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:45.361988 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45213: BlockingRecv error: recv got EOF from 127.0.98.3:45213 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 12:30:45 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:45 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:45.964435 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:45.958867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.1:37345)
0619 12:30:45.959144 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:45.959158 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:45.959253 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:45.959502 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:45.959504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:45.959684 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:45.959778 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:45.959783 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:45.960558 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:45.960561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:45.961182 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:45.961191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:45.961611 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:45.961614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:45.961675 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:45.962179 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:45.962190 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:45.964261 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":90,"threads_started":1}
W20260619 12:30:45.964538 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:45.958938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37345 (local address 127.0.98.62:43419)
0619 12:30:45.959176 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:45.959180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:45.959210 (+ 30us) server_negotiation.cc:407] Connection header received
0619 12:30:45.959336 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:45.959338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:45.959383 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:45.959464 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:45.959906 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:45.960451 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:45.961293 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:45.961514 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:45.961594 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:45.964400 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37345: BlockingRecv error: recv got EOF from 127.0.98.1:37345 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
Jun 19 12:30:46 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:46 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:46.181018 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:46.175367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.2:34429)
0619 12:30:46.175673 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:46.175688 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:46.175831 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:46.176091 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:46.176094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:46.176302 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:46.176396 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:46.176402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:46.177237 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:46.177242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:46.177809 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:46.177816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:46.178287 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:46.178290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:46.178377 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:46.178965 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:46.178978 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:46.180848 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":126,"threads_started":1}
W20260619 12:30:46.181075 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:46.175678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34429 (local address 127.0.98.62:43419)
0619 12:30:46.175828 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:46.175833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:46.175848 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:30:46.175918 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:46.175922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:46.175978 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:46.176070 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:46.176520 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:46.177049 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:46.177934 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:46.178164 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:46.178282 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:46.180969 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34429: BlockingRecv error: recv got EOF from 127.0.98.2:34429 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 12:30:46 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:46 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:46.368510 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:46.362840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.3:48055)
0619 12:30:46.363127 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:46.363140 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:46.363254 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:46.363571 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:46.363574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:46.363757 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:46.363851 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:46.363856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:46.364646 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:46.364649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:46.365252 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:46.365262 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:46.365762 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:46.365767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:46.365861 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:46.366424 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:46.366436 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:46.368397 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":113,"threads_started":1}
W20260619 12:30:46.368677 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:46.362971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48055 (local address 127.0.98.62:43419)
0619 12:30:46.363116 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:46.363119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:46.363228 (+ 109us) server_negotiation.cc:407] Connection header received
0619 12:30:46.363393 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:46.363396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:46.363447 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:46.363527 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:46.363965 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:46.364498 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:46.365381 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:46.365624 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:46.365718 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:46.368548 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48055: BlockingRecv error: recv got EOF from 127.0.98.3:48055 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 12:30:46 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:46 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:46.970779 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:46.965192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.1:35127)
0619 12:30:46.965477 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:46.965496 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:46.965605 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:46.965849 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:46.965851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:46.966036 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:46.966133 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:46.966138 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:46.966868 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:46.966871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:46.967470 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:46.967479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:46.967953 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:46.967956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:46.968022 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:46.968534 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:46.968546 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:46.970657 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":66,"threads_started":1}
W20260619 12:30:46.970993 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:46.965358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35127 (local address 127.0.98.62:43419)
0619 12:30:46.965597 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:46.965601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:46.965613 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:30:46.965689 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:46.965691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:46.965730 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:46.965803 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:46.966235 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:46.966746 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:46.967589 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:46.967810 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:46.967909 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:46.970802 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35127: BlockingRecv error: recv got EOF from 127.0.98.1:35127 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
Jun 19 12:30:47 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:47 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:47.189528 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:47.181925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.2:32903)
0619 12:30:47.182292 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:47.182310 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:47.182440 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:47.182705 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:47.182708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:47.182900 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:47.183001 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:47.183007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:47.183901 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:47.183904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:47.184766 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:47.184778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:47.185434 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:47.185438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:47.185539 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:47.186912 (+ 1373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:47.186930 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:47.189351 (+ 2421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":106,"threads_started":1}
W20260619 12:30:47.189591 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:47.182175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32903 (local address 127.0.98.62:43419)
0619 12:30:47.182337 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:47.182341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:47.182376 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:30:47.182534 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:47.182537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:47.182580 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:47.182649 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:47.183119 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:47.183769 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:47.184924 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:47.185244 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:47.185400 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:47.189482 (+ 4082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32903: BlockingRecv error: recv got EOF from 127.0.98.2:32903 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 19 12:30:47 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:47 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:47.374984 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:47.369367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.3:39431)
0619 12:30:47.369636 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:47.369649 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:47.369775 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:47.370070 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:47.370072 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:47.370269 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:47.370362 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:47.370367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:47.371163 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:47.371166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:47.371732 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:47.371741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:47.372200 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:47.372203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:47.372268 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:47.372850 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:47.372862 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:47.374811 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":69,"threads_started":1}
W20260619 12:30:47.375043 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:47.369496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39431 (local address 127.0.98.62:43419)
0619 12:30:47.369650 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:47.369653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:47.369835 (+ 182us) server_negotiation.cc:407] Connection header received
0619 12:30:47.369883 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:47.369885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:47.369933 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:47.370009 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:47.370503 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:47.371036 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:47.371866 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:47.372089 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:47.372174 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:47.374935 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39431: BlockingRecv error: recv got EOF from 127.0.98.3:39431 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 12:30:47 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:47 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:47.977309 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:47.971669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.1:35755)
0619 12:30:47.971969 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:47.971982 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:47.972095 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:47.972335 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:47.972338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:47.972519 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:47.972613 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:47.972618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:47.973438 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:47.973441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:47.974021 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:47.974031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:47.974509 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:47.974512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:47.974576 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:47.975108 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:47.975119 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:47.977188 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":101,"threads_started":1}
W20260619 12:30:47.977502 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:47.971745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35755 (local address 127.0.98.62:43419)
0619 12:30:47.972052 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:47.972055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:47.972070 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:30:47.972181 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:47.972183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:47.972223 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:47.972296 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:47.972721 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:47.973301 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:47.974141 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:47.974373 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:47.974455 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:47.977352 (+ 2897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35755: BlockingRecv error: recv got EOF from 127.0.98.1:35755 (error 108)
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
Jun 19 12:30:48 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:48 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:48.196875 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:48.190364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.2:59939)
0619 12:30:48.190679 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:48.190696 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:48.190823 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:48.191239 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:48.191243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:48.191515 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:48.191657 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:48.191666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:48.192671 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:48.192675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:48.193468 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:48.193478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:48.194104 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:48.194108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:48.194199 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:48.194791 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:48.194822 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:48.196706 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":91,"threads_started":1}
W20260619 12:30:48.196974 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:48.190460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59939 (local address 127.0.98.62:43419)
0619 12:30:48.190601 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:48.190606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:48.190784 (+ 178us) server_negotiation.cc:407] Connection header received
0619 12:30:48.190956 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:48.190961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:48.191016 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:48.191088 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:48.191804 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:48.192549 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:48.193609 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:48.193965 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:48.194049 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:48.196865 (+ 2816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59939: BlockingRecv error: recv got EOF from 127.0.98.2:59939 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 12:30:48 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:48 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:48.381585 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:48.375845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.3:51063)
0619 12:30:48.376187 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:48.376200 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:48.376341 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:48.376658 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:48.376661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:48.376862 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:48.376975 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:48.376981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:48.377885 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:48.377888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:48.378455 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:48.378462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:48.378901 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:48.378905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:48.378999 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:48.379577 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:48.379588 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:48.381429 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
W20260619 12:30:48.381700 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:48.375971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51063 (local address 127.0.98.62:43419)
0619 12:30:48.376136 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:48.376141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:48.376279 (+ 138us) server_negotiation.cc:407] Connection header received
0619 12:30:48.376495 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:48.376497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:48.376543 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:48.376632 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:48.377125 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:48.377763 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:48.378569 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:48.378791 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:48.378899 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:48.381591 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51063: BlockingRecv error: recv got EOF from 127.0.98.3:51063 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 12:30:48 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:48 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:48.984218 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:48.978428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.1:51867)
0619 12:30:48.978725 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:48.978743 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:48.978895 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:48.979160 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:48.979170 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:48.979404 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:48.979531 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:48.979541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:48.980434 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:48.980437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:48.980996 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:48.981003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:48.981481 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:48.981484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:48.981546 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:48.982108 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:48.982119 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:48.984035 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":84,"threads_started":1}
W20260619 12:30:48.984277 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:48.978531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51867 (local address 127.0.98.62:43419)
0619 12:30:48.978784 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:48.978788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:48.978886 (+ 98us) server_negotiation.cc:407] Connection header received
0619 12:30:48.978971 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:48.978976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:48.979033 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:48.979139 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:48.979758 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:48.980298 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:48.981122 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:48.981379 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:48.981463 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:48.984154 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51867: BlockingRecv error: recv got EOF from 127.0.98.1:51867 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":58,"threads_started":1}
Jun 19 12:30:49 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 12:30:49 dist-test-slave-fg3h krb5kdc[8561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 12:30:49.207033 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:49.197895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43419 (local address 127.0.98.2:42577)
0619 12:30:49.198188 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:49.198201 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:49.198289 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:49.198492 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:49.198494 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:49.198729 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:49.198827 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:49.198832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:49.199833 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:49.199837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:49.200410 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:49.200421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:49.200923 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:49.200925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:49.200986 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:49.204855 (+ 3869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:49.204869 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:49.206850 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43419: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":112,"threads_started":1}
W20260619 12:30:49.207068 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:49.198042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42577 (local address 127.0.98.62:43419)
0619 12:30:49.198231 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:49.198235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:49.198249 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:30:49.198361 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:49.198363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:49.198406 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:49.198478 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:49.198954 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:49.199554 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:49.200540 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:49.200790 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:49.200880 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:49.206965 (+ 6085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42577: BlockingRecv error: recv got EOF from 127.0.98.2:42577 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
I20260619 12:30:49.271972 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 9144
I20260619 12:30:49.277765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 9296
I20260619 12:30:49.283366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 9429
I20260619 12:30:49.288717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 9074
2026-06-19T12:30:49Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18011 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T12:30:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:30:49Z Disabled control of system clock
I20260619 12:30:49.319531 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38773
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 12:30:49.426822 9624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:49.427073 9624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:49.427104 9624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:49.431048 9624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:49.431105 9624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:49.431121 9624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:49.431137 9624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:49.431154 9624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:30:49.435953 9624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38773
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:49.436949 9624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:49.438319 9624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:49.446786 9632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:49.446786 9629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:49.447019 9624 server_base.cc:1034] running on GCE node
W20260619 12:30:49.446789 9630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:49.447759 9624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:49.448876 9624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:49.450071 9624 hybrid_clock.cc:648] HybridClock initialized: now 1781872249450060 us; error 37 us; skew 500 ppm
I20260619 12:30:49.451903 9624 webserver.cc:460] Webserver started at http://127.0.98.62:44063/ using document root <none> and password file <none>
I20260619 12:30:49.452477 9624 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:49.452569 9624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:49.452826 9624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:49.454936 9624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "93ba8eb3550943a5ab3dfa13388b5ad5"
format_stamp: "Formatted at 2026-06-19 12:30:49 on dist-test-slave-fg3h"
server_key: "0f41b4522c5e89d3741d09ea776f361e"
server_key_iv: "a1390e44f62aae66061eb871bf008994"
server_key_version: "encryptionkey@0"
I20260619 12:30:49.455506 9624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "93ba8eb3550943a5ab3dfa13388b5ad5"
format_stamp: "Formatted at 2026-06-19 12:30:49 on dist-test-slave-fg3h"
server_key: "0f41b4522c5e89d3741d09ea776f361e"
server_key_iv: "a1390e44f62aae66061eb871bf008994"
server_key_version: "encryptionkey@0"
I20260619 12:30:49.459215 9624 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 12:30:49.461841 9638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:49.463063 9624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:49.463204 9624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "93ba8eb3550943a5ab3dfa13388b5ad5"
format_stamp: "Formatted at 2026-06-19 12:30:49 on dist-test-slave-fg3h"
server_key: "0f41b4522c5e89d3741d09ea776f361e"
server_key_iv: "a1390e44f62aae66061eb871bf008994"
server_key_version: "encryptionkey@0"
I20260619 12:30:49.463320 9624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:49.481705 9624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:49.482409 9624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:49.482613 9624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:49.489879 9624 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38773
I20260619 12:30:49.489903 9690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38773 every 8 connection(s)
I20260619 12:30:49.490789 9624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:30:49.493848 9691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:49.495296 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 9624
I20260619 12:30:49.495434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:30:49.495729 392 external_mini_cluster.cc:1428] Setting key 256b9e780674a3f95e3723c05d451c34
I20260619 12:30:49.500142 9691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5: Bootstrap starting.
I20260619 12:30:49.502475 9691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:49.503353 9691 log.cc:824] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:49.505234 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:49.497160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59690 (local address 127.0.98.62:38773)
0619 12:30:49.497804 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:49.497818 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:49.497863 (+ 45us) server_negotiation.cc:407] Connection header received
0619 12:30:49.498523 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:49.498539 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:49.498936 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:49.499340 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:49.499741 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:49.500742 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:49.501677 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:49.502150 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:49.502336 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:49.503284 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:49.503289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:49.503302 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:49.503751 (+ 449us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:49.503820 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:49.503846 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:49.503959 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:49.504135 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 12:30:49.504456 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":131,"threads_started":1}
I20260619 12:30:49.505496 9691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5: No bootstrap required, opened a new log
I20260619 12:30:49.508894 9691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ba8eb3550943a5ab3dfa13388b5ad5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38773 } }
I20260619 12:30:49.509159 9691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:49.509243 9691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93ba8eb3550943a5ab3dfa13388b5ad5, State: Initialized, Role: FOLLOWER
I20260619 12:30:49.509682 9691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ba8eb3550943a5ab3dfa13388b5ad5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38773 } }
I20260619 12:30:49.509842 9691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:30:49.509933 9691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:30:49.510030 9691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:49.510949 9691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ba8eb3550943a5ab3dfa13388b5ad5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38773 } }
I20260619 12:30:49.511293 9691 leader_election.cc:302] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93ba8eb3550943a5ab3dfa13388b5ad5; no voters:
I20260619 12:30:49.511600 9691 leader_election.cc:288] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:30:49.511731 9696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:49.511971 9696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5 [term 1 LEADER]: Becoming Leader. State: Replica: 93ba8eb3550943a5ab3dfa13388b5ad5, State: Running, Role: LEADER
I20260619 12:30:49.512308 9696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ba8eb3550943a5ab3dfa13388b5ad5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38773 } }
I20260619 12:30:49.512923 9691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:49.514042 9698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93ba8eb3550943a5ab3dfa13388b5ad5. Latest consensus state: current_term: 1 leader_uuid: "93ba8eb3550943a5ab3dfa13388b5ad5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ba8eb3550943a5ab3dfa13388b5ad5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38773 } } }
I20260619 12:30:49.514175 9698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:49.514510 9697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93ba8eb3550943a5ab3dfa13388b5ad5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ba8eb3550943a5ab3dfa13388b5ad5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38773 } } }
I20260619 12:30:49.514611 9697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5 [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:49.514549 9705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:30:49.517329 9705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:30:49.522557 9705 catalog_manager.cc:1347] Generated new cluster ID: 1413cce7738c4e9d9fa24e78475f82db
I20260619 12:30:49.522630 9705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:30:49.527207 9705 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:30:49.528052 9705 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:30:49.531652 9705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93ba8eb3550943a5ab3dfa13388b5ad5: Generated new TSK 0
I20260619 12:30:49.532290 9705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:30:49.535645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38773
--builtin_ntp_servers=127.0.98.20:40779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 12:30:49.650040 9715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:49.650292 9715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:49.650329 9715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:49.654486 9715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:49.654538 9715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:49.654623 9715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:30:49.659457 9715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:49.660518 9715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:49.661966 9715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:49.671803 9721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:49.671813 9723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:49.671912 9715 server_base.cc:1034] running on GCE node
W20260619 12:30:49.671801 9720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:49.672514 9715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:49.673179 9715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:49.674340 9715 hybrid_clock.cc:648] HybridClock initialized: now 1781872249674324 us; error 27 us; skew 500 ppm
I20260619 12:30:49.676187 9715 webserver.cc:460] Webserver started at http://127.0.98.1:35981/ using document root <none> and password file <none>
I20260619 12:30:49.676791 9715 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:49.676855 9715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:49.677140 9715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:49.679149 9715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "f95dd48011df41f2b097e6ea4074f115"
format_stamp: "Formatted at 2026-06-19 12:30:49 on dist-test-slave-fg3h"
server_key: "033d928eb5f68444c4125f16bf60eb9e"
server_key_iv: "9f96efe24763935a46642504fddd960b"
server_key_version: "encryptionkey@0"
I20260619 12:30:49.679724 9715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f95dd48011df41f2b097e6ea4074f115"
format_stamp: "Formatted at 2026-06-19 12:30:49 on dist-test-slave-fg3h"
server_key: "033d928eb5f68444c4125f16bf60eb9e"
server_key_iv: "9f96efe24763935a46642504fddd960b"
server_key_version: "encryptionkey@0"
I20260619 12:30:49.683306 9715 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 12:30:49.686007 9729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:49.687253 9715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:49.687350 9715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "f95dd48011df41f2b097e6ea4074f115"
format_stamp: "Formatted at 2026-06-19 12:30:49 on dist-test-slave-fg3h"
server_key: "033d928eb5f68444c4125f16bf60eb9e"
server_key_iv: "9f96efe24763935a46642504fddd960b"
server_key_version: "encryptionkey@0"
I20260619 12:30:49.687496 9715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:49.709170 9715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:49.709879 9715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:49.710084 9715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:49.710731 9715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:49.711709 9715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:49.711776 9715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:49.711840 9715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:49.711894 9715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:49.720534 9715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40779
I20260619 12:30:49.720609 9842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40779 every 8 connection(s)
I20260619 12:30:49.721477 9715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:30:49.725186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 9715
I20260619 12:30:49.725302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:30:49.725672 392 external_mini_cluster.cc:1428] Setting key 2917b8a49fdcae6eee38753c954ac1b4
I20260619 12:30:49.727720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38773
--builtin_ntp_servers=127.0.98.20:40779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 12:30:49.731707 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:49.723327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44393 (local address 127.0.98.62:38773)
0619 12:30:49.723490 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:49.723496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:49.724568 (+ 1072us) server_negotiation.cc:407] Connection header received
0619 12:30:49.726198 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:49.726201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:49.726263 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:49.726362 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:49.727451 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:49.728298 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:49.729506 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:49.729891 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:49.730015 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:49.731071 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:49.731078 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:49.731084 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:49.731101 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:49.731127 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:49.731142 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:49.731227 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:49.731412 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 12:30:49.731491 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 12:30:49.732439 9845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:49.723763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38773 (local address 127.0.98.1:44393)
0619 12:30:49.724421 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:49.724462 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:49.725939 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:49.726559 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:49.726568 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:49.726668 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:49.727307 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:49.727321 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:49.728431 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:49.728434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:49.729376 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:49.729385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:49.730040 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:49.730043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:49.730184 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:49.730808 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:30:49.730809 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:49.730847 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:30:49.730916 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:49.731248 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:49.731256 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:49.731262 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 12:30:49.731384 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 12:30:49.731606 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":116,"threads_started":1}
I20260619 12:30:49.733817 9843 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38773
I20260619 12:30:49.734086 9843 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:49.734762 9843 heartbeater.cc:511] Master 127.0.98.62:38773 requested a full tablet report, sending...
I20260619 12:30:49.736349 9655 ts_manager.cc:194] Registered new tserver with Master: f95dd48011df41f2b097e6ea4074f115 (127.0.98.1:40779)
I20260619 12:30:49.737574 9655 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:44393
W20260619 12:30:49.839107 9846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:49.839387 9846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:49.839424 9846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:49.843552 9846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:49.843608 9846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:49.843710 9846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:30:49.848600 9846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:49.849779 9846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:49.851218 9846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:49.861130 9851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:49.861132 9852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:49.861235 9854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:49.861564 9846 server_base.cc:1034] running on GCE node
I20260619 12:30:49.861974 9846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:49.862612 9846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:49.863799 9846 hybrid_clock.cc:648] HybridClock initialized: now 1781872249863778 us; error 61 us; skew 500 ppm
I20260619 12:30:49.865686 9846 webserver.cc:460] Webserver started at http://127.0.98.2:34817/ using document root <none> and password file <none>
I20260619 12:30:49.866269 9846 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:49.866361 9846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:49.866595 9846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:49.868678 9846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "ea21b7731f99478584b818da2d40fd09"
format_stamp: "Formatted at 2026-06-19 12:30:49 on dist-test-slave-fg3h"
server_key: "366822dd2c3ca58a285781a6f474ab2c"
server_key_iv: "9d17ede27a6913e6ab83ec012d7bbcb7"
server_key_version: "encryptionkey@0"
I20260619 12:30:49.869295 9846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ea21b7731f99478584b818da2d40fd09"
format_stamp: "Formatted at 2026-06-19 12:30:49 on dist-test-slave-fg3h"
server_key: "366822dd2c3ca58a285781a6f474ab2c"
server_key_iv: "9d17ede27a6913e6ab83ec012d7bbcb7"
server_key_version: "encryptionkey@0"
I20260619 12:30:49.872968 9846 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 12:30:49.875787 9860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:49.877192 9846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:49.877363 9846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "ea21b7731f99478584b818da2d40fd09"
format_stamp: "Formatted at 2026-06-19 12:30:49 on dist-test-slave-fg3h"
server_key: "366822dd2c3ca58a285781a6f474ab2c"
server_key_iv: "9d17ede27a6913e6ab83ec012d7bbcb7"
server_key_version: "encryptionkey@0"
I20260619 12:30:49.877490 9846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:49.893934 9846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:49.894629 9846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:49.894833 9846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:49.895661 9846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:49.896669 9846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:49.896742 9846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:49.896811 9846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:49.896860 9846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:49.905661 9846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32951
I20260619 12:30:49.905691 9973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32951 every 8 connection(s)
I20260619 12:30:49.906528 9846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:30:49.914206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 9846
I20260619 12:30:49.914324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:30:49.914691 392 external_mini_cluster.cc:1428] Setting key 1c4208f706168fa0027dab8cde5e8106
I20260619 12:30:49.915928 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:49.908217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34665 (local address 127.0.98.62:38773)
0619 12:30:49.908434 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:49.908439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:49.909477 (+ 1038us) server_negotiation.cc:407] Connection header received
0619 12:30:49.910846 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:49.910849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:49.910894 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:49.910971 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:49.911877 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:49.912405 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:49.913602 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:49.913839 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:49.914030 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:49.915435 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:49.915439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:49.915441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:49.915453 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:49.915474 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:49.915485 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:49.915579 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:49.915747 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 12:30:49.915819 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 12:30:49.916932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38773
--builtin_ntp_servers=127.0.98.20:40779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 12:30:49.916971 9976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:49.908648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38773 (local address 127.0.98.2:34665)
0619 12:30:49.909334 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:49.909373 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:49.910652 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:49.911124 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:49.911131 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:49.911211 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:49.911739 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:49.911753 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:49.912508 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:49.912510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:49.913452 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:49.913461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:49.914078 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:49.914084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:49.914308 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:49.915135 (+ 827us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:30:49.915138 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:49.915190 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:30:49.915279 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:49.915592 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:49.915600 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:49.915605 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:30:49.915729 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 12:30:49.915997 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":128,"threads_started":1}
I20260619 12:30:49.918167 9974 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38773
I20260619 12:30:49.918470 9974 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:49.919183 9974 heartbeater.cc:511] Master 127.0.98.62:38773 requested a full tablet report, sending...
I20260619 12:30:49.920348 9655 ts_manager.cc:194] Registered new tserver with Master: ea21b7731f99478584b818da2d40fd09 (127.0.98.2:32951)
I20260619 12:30:49.921007 9655 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34665
W20260619 12:30:50.036283 9977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:50.036568 9977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:50.036607 9977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:50.040769 9977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:50.040827 9977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:50.040917 9977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:30:50.045790 9977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:50.046857 9977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:50.048249 9977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:50.057510 9983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:50.057654 9977 server_base.cc:1034] running on GCE node
W20260619 12:30:50.057518 9982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:50.057614 9985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:50.058189 9977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:50.058861 9977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:50.060068 9977 hybrid_clock.cc:648] HybridClock initialized: now 1781872250060054 us; error 34 us; skew 500 ppm
I20260619 12:30:50.062206 9977 webserver.cc:460] Webserver started at http://127.0.98.3:44381/ using document root <none> and password file <none>
I20260619 12:30:50.062785 9977 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:50.062880 9977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:50.063115 9977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:50.065299 9977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "9f3d3f33edc14f60935aa4e42a347fcf"
format_stamp: "Formatted at 2026-06-19 12:30:50 on dist-test-slave-fg3h"
server_key: "f5a197c3d18ac14539297b88ea896829"
server_key_iv: "225d606efd2b53f2fd593d335a952ca7"
server_key_version: "encryptionkey@0"
I20260619 12:30:50.065867 9977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9f3d3f33edc14f60935aa4e42a347fcf"
format_stamp: "Formatted at 2026-06-19 12:30:50 on dist-test-slave-fg3h"
server_key: "f5a197c3d18ac14539297b88ea896829"
server_key_iv: "225d606efd2b53f2fd593d335a952ca7"
server_key_version: "encryptionkey@0"
I20260619 12:30:50.069705 9977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:30:50.072222 9991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:50.073562 9977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:50.073693 9977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "9f3d3f33edc14f60935aa4e42a347fcf"
format_stamp: "Formatted at 2026-06-19 12:30:50 on dist-test-slave-fg3h"
server_key: "f5a197c3d18ac14539297b88ea896829"
server_key_iv: "225d606efd2b53f2fd593d335a952ca7"
server_key_version: "encryptionkey@0"
I20260619 12:30:50.073807 9977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:50.102706 9977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:50.103466 9977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:50.103672 9977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:50.104297 9977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:50.105289 9977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:50.105363 9977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:50.105435 9977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:50.105477 9977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:50.114521 9977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40665
I20260619 12:30:50.114573 10104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40665 every 8 connection(s)
I20260619 12:30:50.115475 9977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:30:50.124465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 9977
I20260619 12:30:50.124571 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:30:50.124881 392 external_mini_cluster.cc:1428] Setting key df8bbde9fba0eb6f130351a2c0a34203
I20260619 12:30:50.125401 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:50.117349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49523 (local address 127.0.98.62:38773)
0619 12:30:50.117487 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:50.117491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:50.118598 (+ 1107us) server_negotiation.cc:407] Connection header received
0619 12:30:50.120028 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:50.120031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:50.120078 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:50.120157 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:50.121036 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:50.121629 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:50.123170 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:50.123436 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:50.123534 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:50.124812 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:50.124817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:50.124839 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:50.124859 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:50.124879 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:50.124887 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:50.124969 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:50.125193 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 12:30:50.125256 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 12:30:50.126467 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:50.117813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38773 (local address 127.0.98.3:49523)
0619 12:30:50.118447 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:50.118487 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:50.119813 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:50.120292 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:50.120298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:50.120383 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:50.120895 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:50.120909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:50.121764 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:50.121767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:50.122692 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:50.122702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:50.123616 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:50.123620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:50.123783 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:50.124481 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:30:50.124483 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:50.124547 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:30:50.124647 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:50.124982 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:50.124994 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:50.125001 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 12:30:50.125192 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 12:30:50.125508 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
I20260619 12:30:50.127544 10105 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38773
I20260619 12:30:50.127786 10105 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:50.128309 10105 heartbeater.cc:511] Master 127.0.98.62:38773 requested a full tablet report, sending...
I20260619 12:30:50.129339 9655 ts_manager.cc:194] Registered new tserver with Master: 9f3d3f33edc14f60935aa4e42a347fcf (127.0.98.3:40665)
I20260619 12:30:50.129937 9655 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:49523
I20260619 12:30:50.138839 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 12:30:50.141851 9694 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:50.140747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59706 (local address 127.0.98.62:38773)
0619 12:30:50.140937 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:50.140941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:50.141063 (+ 122us) server_negotiation.cc:407] Connection header received
0619 12:30:50.141260 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:50.141264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:50.141315 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:50.141399 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:50.141752 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59706: BlockingRecv error: recv got EOF from 127.0.0.1:59706 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 12:30:50.142330 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 9715
I20260619 12:30:50.147733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 9846
I20260619 12:30:50.152976 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 9977
I20260619 12:30:50.158195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 9624
2026-06-19T12:30:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (867 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T12:30:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:30:50Z Disabled control of system clock
I20260619 12:30:50.187316 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39039
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 12:30:50.301585 10116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:50.301855 10116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:50.301893 10116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:50.306026 10116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:50.306097 10116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:50.306113 10116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:50.306131 10116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:50.306146 10116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:30:50.310874 10116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39039
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:50.311904 10116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:50.313385 10116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:50.321743 10121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:50.321892 10116 server_base.cc:1034] running on GCE node
W20260619 12:30:50.321842 10122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:50.322131 10124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:50.322789 10116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:50.323885 10116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:50.325134 10116 hybrid_clock.cc:648] HybridClock initialized: now 1781872250325069 us; error 34 us; skew 500 ppm
I20260619 12:30:50.327009 10116 webserver.cc:460] Webserver started at http://127.0.98.62:33523/ using document root <none> and password file <none>
I20260619 12:30:50.327798 10116 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:50.327913 10116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:50.328150 10116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:50.330344 10116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "745e74f5257a49dd8eba93b01db9b1fb"
format_stamp: "Formatted at 2026-06-19 12:30:50 on dist-test-slave-fg3h"
server_key: "17f6e8153b0721ac538524e97238d5aa"
server_key_iv: "1c30a57ae2c42d848c565a21307c215c"
server_key_version: "encryptionkey@0"
I20260619 12:30:50.330916 10116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "745e74f5257a49dd8eba93b01db9b1fb"
format_stamp: "Formatted at 2026-06-19 12:30:50 on dist-test-slave-fg3h"
server_key: "17f6e8153b0721ac538524e97238d5aa"
server_key_iv: "1c30a57ae2c42d848c565a21307c215c"
server_key_version: "encryptionkey@0"
I20260619 12:30:50.334738 10116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:30:50.337186 10130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:50.338420 10116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:30:50.338562 10116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "745e74f5257a49dd8eba93b01db9b1fb"
format_stamp: "Formatted at 2026-06-19 12:30:50 on dist-test-slave-fg3h"
server_key: "17f6e8153b0721ac538524e97238d5aa"
server_key_iv: "1c30a57ae2c42d848c565a21307c215c"
server_key_version: "encryptionkey@0"
I20260619 12:30:50.338678 10116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:50.343276 10116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:50.343855 10116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:50.344034 10116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:50.350953 10116 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39039
I20260619 12:30:50.350966 10182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39039 every 8 connection(s)
I20260619 12:30:50.351850 10116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:30:50.353315 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 10116
I20260619 12:30:50.353413 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:30:50.353714 392 external_mini_cluster.cc:1428] Setting key 3ddcc23f112d0b8679af0ec35812ff80
I20260619 12:30:50.355378 10183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:50.359139 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:50.355029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58304 (local address 127.0.98.62:39039)
0619 12:30:50.355676 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:50.355691 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:50.355726 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:30:50.356356 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:50.356414 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:50.356753 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:50.357033 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:50.357875 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:50.357880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:50.357891 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:50.357901 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:50.357951 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:50.357966 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:50.358075 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:50.358207 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 12:30:50.358496 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":145,"threads_started":1}
I20260619 12:30:50.361598 10183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb: Bootstrap starting.
I20260619 12:30:50.363696 10183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:50.364487 10183 log.cc:824] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:50.366591 10183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb: No bootstrap required, opened a new log
I20260619 12:30:50.369908 10183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745e74f5257a49dd8eba93b01db9b1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39039 } }
I20260619 12:30:50.370142 10183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:50.370221 10183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745e74f5257a49dd8eba93b01db9b1fb, State: Initialized, Role: FOLLOWER
I20260619 12:30:50.370632 10183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745e74f5257a49dd8eba93b01db9b1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39039 } }
I20260619 12:30:50.370779 10183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:30:50.370862 10183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:30:50.370977 10183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:50.371822 10183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745e74f5257a49dd8eba93b01db9b1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39039 } }
I20260619 12:30:50.372171 10183 leader_election.cc:302] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 745e74f5257a49dd8eba93b01db9b1fb; no voters:
I20260619 12:30:50.372550 10183 leader_election.cc:288] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:30:50.372603 10188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:50.372783 10188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb [term 1 LEADER]: Becoming Leader. State: Replica: 745e74f5257a49dd8eba93b01db9b1fb, State: Running, Role: LEADER
I20260619 12:30:50.373066 10188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745e74f5257a49dd8eba93b01db9b1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39039 } }
I20260619 12:30:50.373823 10183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:50.374881 10190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "745e74f5257a49dd8eba93b01db9b1fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745e74f5257a49dd8eba93b01db9b1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39039 } } }
I20260619 12:30:50.375025 10190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:50.375329 10189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 745e74f5257a49dd8eba93b01db9b1fb. Latest consensus state: current_term: 1 leader_uuid: "745e74f5257a49dd8eba93b01db9b1fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745e74f5257a49dd8eba93b01db9b1fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39039 } } }
I20260619 12:30:50.375425 10189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:50.375536 10197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:30:50.378639 10197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:30:50.384186 10197 catalog_manager.cc:1347] Generated new cluster ID: dc09f1580cae499188a39d785a7fea38
I20260619 12:30:50.384258 10197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:30:50.411019 10197 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:30:50.411403 10197 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:30:50.428153 10197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 745e74f5257a49dd8eba93b01db9b1fb: Generated new TSK 0
I20260619 12:30:50.429064 10197 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:30:50.440740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39039
--builtin_ntp_servers=127.0.98.20:37323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 12:30:50.560753 10207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:50.561076 10207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:50.561173 10207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:50.565537 10207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:50.565624 10207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:50.565714 10207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:30:50.570528 10207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:50.571651 10207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:50.573062 10207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:50.583164 10212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:50.583201 10207 server_base.cc:1034] running on GCE node
W20260619 12:30:50.583196 10215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:50.583417 10213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:50.583815 10207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:50.584486 10207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:50.585672 10207 hybrid_clock.cc:648] HybridClock initialized: now 1781872250585638 us; error 48 us; skew 500 ppm
I20260619 12:30:50.587682 10207 webserver.cc:460] Webserver started at http://127.0.98.1:32953/ using document root <none> and password file <none>
I20260619 12:30:50.588271 10207 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:50.588374 10207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:50.588605 10207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:50.590727 10207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "32ee215aa3a34bb4b692789a12e5ae57"
format_stamp: "Formatted at 2026-06-19 12:30:50 on dist-test-slave-fg3h"
server_key: "0d2f6db622ab87c47a1691bcf374890b"
server_key_iv: "99280f14ecd210c2f7455641cc63c02e"
server_key_version: "encryptionkey@0"
I20260619 12:30:50.591317 10207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "32ee215aa3a34bb4b692789a12e5ae57"
format_stamp: "Formatted at 2026-06-19 12:30:50 on dist-test-slave-fg3h"
server_key: "0d2f6db622ab87c47a1691bcf374890b"
server_key_iv: "99280f14ecd210c2f7455641cc63c02e"
server_key_version: "encryptionkey@0"
I20260619 12:30:50.595204 10207 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:30:50.597838 10221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:50.599243 10207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:30:50.599434 10207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "32ee215aa3a34bb4b692789a12e5ae57"
format_stamp: "Formatted at 2026-06-19 12:30:50 on dist-test-slave-fg3h"
server_key: "0d2f6db622ab87c47a1691bcf374890b"
server_key_iv: "99280f14ecd210c2f7455641cc63c02e"
server_key_version: "encryptionkey@0"
I20260619 12:30:50.599567 10207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:50.604287 10207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:50.604893 10207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:50.605075 10207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:50.605723 10207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:50.606621 10207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:50.606691 10207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:50.606753 10207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:50.606805 10207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:50.615464 10207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46883
I20260619 12:30:50.615492 10334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46883 every 8 connection(s)
I20260619 12:30:50.616338 10207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:30:50.617724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 10207
I20260619 12:30:50.617841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:30:50.618208 392 external_mini_cluster.cc:1428] Setting key 2705479c0881adee503cbb96d95ea321
I20260619 12:30:50.620124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39039
--builtin_ntp_servers=127.0.98.20:37323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 12:30:50.623137 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:50.618335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39793 (local address 127.0.98.62:39039)
0619 12:30:50.618484 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:50.618489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:50.619752 (+ 1263us) server_negotiation.cc:407] Connection header received
0619 12:30:50.621127 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:50.621133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:50.621198 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:50.621302 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:50.622647 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:50.622651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:50.622654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:50.622667 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:50.622691 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:50.622700 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:50.622800 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:50.622953 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 12:30:50.623001 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 12:30:50.623993 10337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:50.618823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39039 (local address 127.0.98.1:39793)
0619 12:30:50.619583 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:50.619635 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:50.620879 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:50.621531 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:50.621544 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:50.621671 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:50.622325 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:30:50.622328 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:50.622386 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:30:50.622484 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:50.622794 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:50.622803 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:50.622811 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 12:30:50.622999 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 12:30:50.623245 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":143,"threads_started":1}
I20260619 12:30:50.625339 10335 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39039
I20260619 12:30:50.625584 10335 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:50.626004 10335 heartbeater.cc:511] Master 127.0.98.62:39039 requested a full tablet report, sending...
I20260619 12:30:50.627609 10147 ts_manager.cc:194] Registered new tserver with Master: 32ee215aa3a34bb4b692789a12e5ae57 (127.0.98.1:46883)
W20260619 12:30:50.733537 10338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:50.733829 10338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:50.733906 10338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:50.737988 10338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:50.738080 10338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:50.738166 10338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:30:50.743129 10338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:50.744220 10338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:50.745643 10338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:50.755589 10344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:50.755605 10343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:50.755605 10346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:50.755668 10338 server_base.cc:1034] running on GCE node
I20260619 12:30:50.756264 10338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:50.756963 10338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:50.758153 10338 hybrid_clock.cc:648] HybridClock initialized: now 1781872250758128 us; error 37 us; skew 500 ppm
I20260619 12:30:50.760006 10338 webserver.cc:460] Webserver started at http://127.0.98.2:40943/ using document root <none> and password file <none>
I20260619 12:30:50.760586 10338 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:50.760679 10338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:50.760905 10338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:50.762995 10338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "10ca96c4da86492c9c5eb13e9baa11cf"
format_stamp: "Formatted at 2026-06-19 12:30:50 on dist-test-slave-fg3h"
server_key: "c339d1c0e60b64221b4e6d7bea73cd20"
server_key_iv: "c586e32798c2dafe01c082cd9c66b39b"
server_key_version: "encryptionkey@0"
I20260619 12:30:50.763602 10338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "10ca96c4da86492c9c5eb13e9baa11cf"
format_stamp: "Formatted at 2026-06-19 12:30:50 on dist-test-slave-fg3h"
server_key: "c339d1c0e60b64221b4e6d7bea73cd20"
server_key_iv: "c586e32798c2dafe01c082cd9c66b39b"
server_key_version: "encryptionkey@0"
I20260619 12:30:50.768006 10338 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 12:30:50.770627 10352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:50.771881 10338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:30:50.772033 10338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "10ca96c4da86492c9c5eb13e9baa11cf"
format_stamp: "Formatted at 2026-06-19 12:30:50 on dist-test-slave-fg3h"
server_key: "c339d1c0e60b64221b4e6d7bea73cd20"
server_key_iv: "c586e32798c2dafe01c082cd9c66b39b"
server_key_version: "encryptionkey@0"
I20260619 12:30:50.772159 10338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:50.777051 10338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:50.777702 10338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:50.777902 10338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:50.778473 10338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:50.779452 10338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:50.779523 10338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:50.779589 10338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:50.779641 10338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:50.789574 10338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35727
I20260619 12:30:50.789597 10465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35727 every 8 connection(s)
I20260619 12:30:50.790536 10338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:30:50.796952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 10338
I20260619 12:30:50.796975 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:50.792513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36971 (local address 127.0.98.62:39039)
0619 12:30:50.792687 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:50.792691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:50.793827 (+ 1136us) server_negotiation.cc:407] Connection header received
0619 12:30:50.795205 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:50.795208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:50.795253 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:50.795342 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:50.796473 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:50.796477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:50.796480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:50.796493 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:50.796517 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:50.796528 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:50.796644 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:50.796796 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 12:30:50.796841 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 12:30:50.797144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:30:50.797508 392 external_mini_cluster.cc:1428] Setting key e913fbeacc214e0831644751c059e70a
I20260619 12:30:50.797973 10468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:50.792890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39039 (local address 127.0.98.2:36971)
0619 12:30:50.793661 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:50.793703 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:50.794958 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:50.795532 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:50.795539 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:50.795632 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:50.796204 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:30:50.796206 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:50.796246 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:30:50.796313 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:50.796629 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:50.796637 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:50.796642 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:30:50.796826 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 12:30:50.797062 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":174,"threads_started":1}
I20260619 12:30:50.799203 10466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39039
I20260619 12:30:50.799448 10466 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:50.799824 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39039
--builtin_ntp_servers=127.0.98.20:37323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 12:30:50.799916 10466 heartbeater.cc:511] Master 127.0.98.62:39039 requested a full tablet report, sending...
I20260619 12:30:50.801040 10147 ts_manager.cc:194] Registered new tserver with Master: 10ca96c4da86492c9c5eb13e9baa11cf (127.0.98.2:35727)
W20260619 12:30:50.914908 10469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:50.915190 10469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:50.915231 10469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:50.919376 10469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:50.919433 10469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:50.919492 10469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:30:50.924288 10469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:50.925357 10469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:50.926836 10469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:50.936393 10477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:50.936362 10475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:50.936362 10474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:50.937184 10469 server_base.cc:1034] running on GCE node
I20260619 12:30:50.937628 10469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:50.938273 10469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:50.939481 10469 hybrid_clock.cc:648] HybridClock initialized: now 1781872250939451 us; error 44 us; skew 500 ppm
I20260619 12:30:50.941471 10469 webserver.cc:460] Webserver started at http://127.0.98.3:43939/ using document root <none> and password file <none>
I20260619 12:30:50.942093 10469 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:50.942189 10469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:50.942435 10469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:50.944578 10469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "0bd2ad1dc6fd4904830cc039dddd04d6"
format_stamp: "Formatted at 2026-06-19 12:30:50 on dist-test-slave-fg3h"
server_key: "a2742f375e34a561670b9916bbfd4897"
server_key_iv: "aa16c1c41f363e68ce057f215afcddc3"
server_key_version: "encryptionkey@0"
I20260619 12:30:50.945226 10469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0bd2ad1dc6fd4904830cc039dddd04d6"
format_stamp: "Formatted at 2026-06-19 12:30:50 on dist-test-slave-fg3h"
server_key: "a2742f375e34a561670b9916bbfd4897"
server_key_iv: "aa16c1c41f363e68ce057f215afcddc3"
server_key_version: "encryptionkey@0"
I20260619 12:30:50.949003 10469 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:30:50.951649 10483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:50.953141 10469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 12:30:50.953272 10469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "0bd2ad1dc6fd4904830cc039dddd04d6"
format_stamp: "Formatted at 2026-06-19 12:30:50 on dist-test-slave-fg3h"
server_key: "a2742f375e34a561670b9916bbfd4897"
server_key_iv: "aa16c1c41f363e68ce057f215afcddc3"
server_key_version: "encryptionkey@0"
I20260619 12:30:50.953408 10469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:50.958045 10469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:50.958657 10469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:50.958853 10469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:50.959409 10469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:50.960294 10469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:50.960377 10469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:50.960444 10469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:50.960489 10469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:50.969906 10469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40667
I20260619 12:30:50.969928 10596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40667 every 8 connection(s)
I20260619 12:30:50.970880 10469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:30:50.976824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 10469
I20260619 12:30:50.976995 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:30:50.977316 392 external_mini_cluster.cc:1428] Setting key 885e051d741e8f4b4d21b33c91d762bd
I20260619 12:30:50.977911 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:50.972769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54101 (local address 127.0.98.62:39039)
0619 12:30:50.972918 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:50.972921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:50.974077 (+ 1156us) server_negotiation.cc:407] Connection header received
0619 12:30:50.975795 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:50.975798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:50.975845 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:50.975953 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:50.977341 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:50.977345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:50.977348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:30:50.977361 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:50.977387 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:30:50.977397 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:50.977490 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:50.977710 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 12:30:50.977752 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 12:30:50.978873 10599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:50.973254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39039 (local address 127.0.98.3:54101)
0619 12:30:50.973914 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:50.973955 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:50.975538 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:50.976159 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:50.976170 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:50.976277 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:50.976988 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:30:50.976992 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:50.977071 (+ 79us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:30:50.977188 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:50.977511 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:50.977522 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:50.977528 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 12:30:50.977732 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 12:30:50.978009 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":110,"threads_started":1}
I20260619 12:30:50.979928 10597 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39039
I20260619 12:30:50.980161 10597 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:50.980615 10597 heartbeater.cc:511] Master 127.0.98.62:39039 requested a full tablet report, sending...
I20260619 12:30:50.981689 10146 ts_manager.cc:194] Registered new tserver with Master: 0bd2ad1dc6fd4904830cc039dddd04d6 (127.0.98.3:40667)
I20260619 12:30:50.991536 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 12:30:50.994577 10186 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:30:50.993593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53556 (local address 127.0.98.62:39039)
0619 12:30:50.993755 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:50.993760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:50.993951 (+ 191us) server_negotiation.cc:407] Connection header received
0619 12:30:50.994102 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:50.994105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:50.994143 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:50.994230 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:50.994448 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53556: BlockingRecv error: recv got EOF from 127.0.0.1:53556 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:30:50.994729 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 10207
I20260619 12:30:51.000263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 10338
I20260619 12:30:51.005614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 10469
I20260619 12:30:51.010692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 10116
2026-06-19T12:30:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (852 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:30:51 dist-test-slave-fg3h krb5kdc[10607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:30:51 dist-test-slave-fg3h krb5kdc[10607](info): set up 2 sockets
Jun 19 12:30:51 dist-test-slave-fg3h krb5kdc[10607](info): commencing operation
krb5kdc: starting...
W20260619 12:30:52.074184 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:30:52 dist-test-slave-fg3h krb5kdc[10607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:30:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:30:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:30:52.200914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33231
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:52.314213 10620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:52.314571 10620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:52.314656 10620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:52.318836 10620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:30:52.318934 10620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:52.318975 10620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:52.319038 10620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:30:52.319078 10620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:30:52.324533 10620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:52.325738 10620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:52.326865 10620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:52.335871 10626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:52.336064 10628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:52.336112 10625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:52.336993 10620 server_base.cc:1034] running on GCE node
I20260619 12:30:52.337713 10620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:52.338848 10620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:52.340029 10620 hybrid_clock.cc:648] HybridClock initialized: now 1781872252340009 us; error 33 us; skew 500 ppm
Jun 19 12:30:52 dist-test-slave-fg3h krb5kdc[10607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:52.343269 10620 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:30:52.344177 10620 webserver.cc:460] Webserver started at http://127.0.98.62:42235/ using document root <none> and password file <none>
I20260619 12:30:52.344717 10620 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:52.344774 10620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:52.344935 10620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:52.347069 10620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "aad94b5232d843fab881b6bbe9b728be"
format_stamp: "Formatted at 2026-06-19 12:30:52 on dist-test-slave-fg3h"
server_key: "1927ec1b79af23196cfff84cf34eb897"
server_key_iv: "dd6254c8bd28b30cd029cc2a97c097fb"
server_key_version: "encryptionkey@0"
I20260619 12:30:52.347640 10620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "aad94b5232d843fab881b6bbe9b728be"
format_stamp: "Formatted at 2026-06-19 12:30:52 on dist-test-slave-fg3h"
server_key: "1927ec1b79af23196cfff84cf34eb897"
server_key_iv: "dd6254c8bd28b30cd029cc2a97c097fb"
server_key_version: "encryptionkey@0"
I20260619 12:30:52.351392 10620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:30:52.353925 10635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:52.355301 10620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:30:52.355513 10620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "aad94b5232d843fab881b6bbe9b728be"
format_stamp: "Formatted at 2026-06-19 12:30:52 on dist-test-slave-fg3h"
server_key: "1927ec1b79af23196cfff84cf34eb897"
server_key_iv: "dd6254c8bd28b30cd029cc2a97c097fb"
server_key_version: "encryptionkey@0"
I20260619 12:30:52.355649 10620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:52.380146 10620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:52.383584 10620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:52.383832 10620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:52.391232 10620 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33231
I20260619 12:30:52.391526 10687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33231 every 8 connection(s)
I20260619 12:30:52.392267 10620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:30:52.395036 10688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:52.397516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 10620
I20260619 12:30:52.397650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:30:52.397951 392 external_mini_cluster.cc:1428] Setting key 330dc6315385093346d5d266d96492bd
I20260619 12:30:52.401365 10688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be: Bootstrap starting.
I20260619 12:30:52.404035 10688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:52.404872 10688 log.cc:824] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:52.406911 10688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be: No bootstrap required, opened a new log
Jun 19 12:30:52 dist-test-slave-fg3h krb5kdc[10607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:52.411057 10688 raft_consensus.cc:348] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad94b5232d843fab881b6bbe9b728be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } }
I20260619 12:30:52.411401 10688 raft_consensus.cc:374] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:52.411469 10688 raft_consensus.cc:729] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad94b5232d843fab881b6bbe9b728be, State: Initialized, Role: FOLLOWER
I20260619 12:30:52.412101 10688 consensus_queue.cc:260] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad94b5232d843fab881b6bbe9b728be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } }
I20260619 12:30:52.412269 10688 raft_consensus.cc:388] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:30:52.412371 10688 raft_consensus.cc:482] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:30:52.412492 10688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:52.413789 10688 raft_consensus.cc:504] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad94b5232d843fab881b6bbe9b728be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } }
I20260619 12:30:52.414186 10688 leader_election.cc:302] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aad94b5232d843fab881b6bbe9b728be; no voters:
I20260619 12:30:52.414466 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:52.399436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44264 (local address 127.0.98.62:33231)
0619 12:30:52.399867 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:52.399885 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:52.399929 (+ 44us) server_negotiation.cc:407] Connection header received
0619 12:30:52.400734 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:52.400757 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:52.401188 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:52.401561 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:52.402249 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:52.403714 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:52.404815 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:52.405493 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:52.405685 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:52.408633 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:52.408638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:52.408651 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:52.408678 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:52.411018 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:52.411609 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:52.411617 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:52.411625 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:52.411749 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:52.412105 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:52.412107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:52.412109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:52.412667 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:52.412804 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:52.413239 (+ 435us) server_negotiation.cc:299] Negotiation successful
0619 12:30:52.413557 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":113,"threads_started":1}
I20260619 12:30:52.414597 10688 leader_election.cc:288] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:30:52.414764 10693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:52.414965 10693 raft_consensus.cc:686] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be [term 1 LEADER]: Becoming Leader. State: Replica: aad94b5232d843fab881b6bbe9b728be, State: Running, Role: LEADER
I20260619 12:30:52.415328 10693 consensus_queue.cc:237] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad94b5232d843fab881b6bbe9b728be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } }
I20260619 12:30:52.416029 10688 sys_catalog.cc:564] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:30:52.417003 10694 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aad94b5232d843fab881b6bbe9b728be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad94b5232d843fab881b6bbe9b728be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } }
I20260619 12:30:52.417155 10694 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:52.417482 10695 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be [sys.catalog]: SysCatalogTable state changed. Reason: New leader aad94b5232d843fab881b6bbe9b728be. Latest consensus state: current_term: 1 leader_uuid: "aad94b5232d843fab881b6bbe9b728be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad94b5232d843fab881b6bbe9b728be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33231 } } }
I20260619 12:30:52.417587 10695 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be [sys.catalog]: This master's current role is: LEADER
I20260619 12:30:52.417850 10702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:30:52.420938 10702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:30:52.426829 10702 catalog_manager.cc:1347] Generated new cluster ID: 293dfcc681e4491ea2385b28fb69a3aa
I20260619 12:30:52.426919 10702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:30:52.432668 10702 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:30:52.433683 10702 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:30:52.439379 10702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be: Generated new TSK 0
I20260619 12:30:52.440027 10702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:30:52.476806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33231
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:52.589371 10714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:52.589624 10714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:52.589668 10714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:52.593672 10714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:52.593727 10714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:52.593786 10714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:30:52.598920 10714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:52.599941 10714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:52.600948 10714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:52.610778 10719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:52.610776 10720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:52.610846 10722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:52.611198 10714 server_base.cc:1034] running on GCE node
I20260619 12:30:52.611634 10714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:52.612308 10714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:52.613597 10714 hybrid_clock.cc:648] HybridClock initialized: now 1781872252613553 us; error 57 us; skew 500 ppm
Jun 19 12:30:52 dist-test-slave-fg3h krb5kdc[10607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:52.616608 10714 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:30:52.617941 10714 webserver.cc:460] Webserver started at http://127.0.98.1:45967/ using document root <none> and password file <none>
I20260619 12:30:52.618810 10714 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:52.618919 10714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:52.619216 10714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:52.622426 10714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "2414cbf8ce09456b88972bce9ce56f84"
format_stamp: "Formatted at 2026-06-19 12:30:52 on dist-test-slave-fg3h"
server_key: "045f2212569748615e253623fc88284c"
server_key_iv: "91187b1106e7b783623ccba9e74aa3a4"
server_key_version: "encryptionkey@0"
I20260619 12:30:52.623227 10714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2414cbf8ce09456b88972bce9ce56f84"
format_stamp: "Formatted at 2026-06-19 12:30:52 on dist-test-slave-fg3h"
server_key: "045f2212569748615e253623fc88284c"
server_key_iv: "91187b1106e7b783623ccba9e74aa3a4"
server_key_version: "encryptionkey@0"
I20260619 12:30:52.627955 10714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:30:52.630566 10729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:52.631866 10714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:30:52.632027 10714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "2414cbf8ce09456b88972bce9ce56f84"
format_stamp: "Formatted at 2026-06-19 12:30:52 on dist-test-slave-fg3h"
server_key: "045f2212569748615e253623fc88284c"
server_key_iv: "91187b1106e7b783623ccba9e74aa3a4"
server_key_version: "encryptionkey@0"
I20260619 12:30:52.632154 10714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:52.658699 10714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:52.662446 10714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:52.662672 10714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:52.663301 10714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:52.664202 10714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:52.664279 10714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:52.664347 10714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:52.664398 10714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:52.673859 10714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41335
I20260619 12:30:52.673895 10842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41335 every 8 connection(s)
I20260619 12:30:52.674779 10714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:30:52.684108 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 10714
I20260619 12:30:52.684235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:30:52.684577 392 external_mini_cluster.cc:1428] Setting key 2e7508387cbd624b740f1c09d6a20266
Jun 19 12:30:52 dist-test-slave-fg3h krb5kdc[10607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:52.689849 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:52.676693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42095 (local address 127.0.98.62:33231)
0619 12:30:52.676857 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:52.676861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:52.677904 (+ 1043us) server_negotiation.cc:407] Connection header received
0619 12:30:52.679250 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:52.679254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:52.679308 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:52.679424 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:52.680625 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:52.681229 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:52.682402 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:52.682637 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:52.682766 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:52.685731 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:52.685734 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:52.685737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:52.685760 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:52.687446 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:52.688072 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:52.688078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:52.688082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:52.688185 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:52.688537 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:52.688542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:52.688543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:52.688786 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:52.688909 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:52.689465 (+ 556us) server_negotiation.cc:299] Negotiation successful
0619 12:30:52.689612 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 12:30:52.690666 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:52.677199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33231 (local address 127.0.98.1:42095)
0619 12:30:52.677646 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:52.677689 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:52.679025 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:52.679599 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:52.679606 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:52.679971 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:52.680463 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:52.680477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:52.681367 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:52.681371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:52.682243 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:52.682252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:52.682806 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:52.682810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:52.682987 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:52.683616 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:52.683632 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:52.685563 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:52.687574 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:52.687582 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:52.687599 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:52.687958 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:52.688287 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:52.688290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:52.688291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:52.688421 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:52.688896 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:52.688900 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:52.689217 (+ 317us) client_negotiation.cc:769] Sending connection context
0619 12:30:52.689390 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 12:30:52.689692 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":98,"threads_started":1}
I20260619 12:30:52.691948 10843 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33231
I20260619 12:30:52.692193 10843 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:52.692703 10843 heartbeater.cc:511] Master 127.0.98.62:33231 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:30:52.694284 10652 ts_manager.cc:194] Registered new tserver with Master: 2414cbf8ce09456b88972bce9ce56f84 (127.0.98.1:41335)
I20260619 12:30:52.695518 10652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:30:52.713428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33231
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:52.827914 10848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:52.828186 10848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:52.828230 10848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:52.832691 10848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:52.832763 10848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:52.832823 10848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:30:52.838523 10848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:52.839715 10848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:52.840781 10848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:30:52.851334 10848 server_base.cc:1034] running on GCE node
W20260619 12:30:52.851258 10854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:52.851298 10856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:52.851277 10853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:52.852066 10848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:52.852723 10848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:52.853907 10848 hybrid_clock.cc:648] HybridClock initialized: now 1781872252853850 us; error 69 us; skew 500 ppm
Jun 19 12:30:52 dist-test-slave-fg3h krb5kdc[10607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:52.856890 10848 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:30:52.857890 10848 webserver.cc:460] Webserver started at http://127.0.98.2:40127/ using document root <none> and password file <none>
I20260619 12:30:52.858529 10848 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:52.858633 10848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:52.858875 10848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:52.861146 10848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "c3db72d6eeed4db7b130b24d3361160f"
format_stamp: "Formatted at 2026-06-19 12:30:52 on dist-test-slave-fg3h"
server_key: "ca6735c0f5f512f1ce7d9177188d92c5"
server_key_iv: "5e4bee391f12e752965ebd3aa44854f6"
server_key_version: "encryptionkey@0"
I20260619 12:30:52.861752 10848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c3db72d6eeed4db7b130b24d3361160f"
format_stamp: "Formatted at 2026-06-19 12:30:52 on dist-test-slave-fg3h"
server_key: "ca6735c0f5f512f1ce7d9177188d92c5"
server_key_iv: "5e4bee391f12e752965ebd3aa44854f6"
server_key_version: "encryptionkey@0"
I20260619 12:30:52.865760 10848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:30:52.868453 10863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:52.870232 10848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:30:52.870383 10848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "c3db72d6eeed4db7b130b24d3361160f"
format_stamp: "Formatted at 2026-06-19 12:30:52 on dist-test-slave-fg3h"
server_key: "ca6735c0f5f512f1ce7d9177188d92c5"
server_key_iv: "5e4bee391f12e752965ebd3aa44854f6"
server_key_version: "encryptionkey@0"
I20260619 12:30:52.870512 10848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:52.895892 10848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:52.899020 10848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:52.899237 10848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:52.899845 10848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:52.900688 10848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:52.900764 10848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:52.900830 10848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:52.900854 10848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:52.909915 10848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45867
I20260619 12:30:52.909933 10976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45867 every 8 connection(s)
I20260619 12:30:52.910802 10848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:30:52.920635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 10848
I20260619 12:30:52.920761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:30:52.921119 392 external_mini_cluster.cc:1428] Setting key e04d1feadfdf38dbe457bb5d32a7b8ef
Jun 19 12:30:52 dist-test-slave-fg3h krb5kdc[10607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:52.927286 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:52.912701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51577 (local address 127.0.98.62:33231)
0619 12:30:52.912842 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:52.912846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:52.913747 (+ 901us) server_negotiation.cc:407] Connection header received
0619 12:30:52.915225 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:52.915229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:52.915292 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:52.915424 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:52.916845 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:52.917709 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:52.918998 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:52.919253 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:52.919403 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:52.922561 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:52.922565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:52.922569 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:52.922605 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:52.924493 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:52.925392 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:52.925396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:52.925398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:52.925486 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:52.925992 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:52.925995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:52.925996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:52.926183 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:52.926269 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:52.926887 (+ 618us) server_negotiation.cc:299] Negotiation successful
0619 12:30:52.927039 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:30:52.927973 10979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:52.913014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33231 (local address 127.0.98.2:51577)
0619 12:30:52.913557 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:52.913602 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:52.914975 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:52.915608 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:52.915616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:52.916015 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:52.916646 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:52.916662 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:52.917857 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:52.917862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:52.918845 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:52.918855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:52.919448 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:52.919452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:52.919639 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:52.920391 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:52.920419 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:52.922375 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:52.924770 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:52.924778 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:52.924796 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:52.925239 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:52.925612 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:52.925616 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:52.925620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:52.925821 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:52.926294 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:52.926299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:52.926597 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 12:30:52.926758 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 12:30:52.927056 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":117,"threads_started":1}
I20260619 12:30:52.929380 10977 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33231
I20260619 12:30:52.929656 10977 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:52.930208 10977 heartbeater.cc:511] Master 127.0.98.62:33231 requested a full tablet report, sending...
I20260619 12:30:52.931329 10652 ts_manager.cc:194] Registered new tserver with Master: c3db72d6eeed4db7b130b24d3361160f (127.0.98.2:45867)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:30:52.931989 10652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:30:52.953238 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33231
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:30:53.067209 10982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:30:53.067510 10982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:30:53.067556 10982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:30:53.072121 10982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:30:53.072186 10982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:30:53.072290 10982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:30:53.078176 10982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:30:53.079356 10982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:30:53.080451 10982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:30:53.090608 10988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:53.090672 10982 server_base.cc:1034] running on GCE node
W20260619 12:30:53.090574 10987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:30:53.090580 10990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:30:53.091243 10982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:30:53.091933 10982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:30:53.093169 10982 hybrid_clock.cc:648] HybridClock initialized: now 1781872253093149 us; error 37 us; skew 500 ppm
Jun 19 12:30:53 dist-test-slave-fg3h krb5kdc[10607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:30:53.096287 10982 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:30:53.097445 10982 webserver.cc:460] Webserver started at http://127.0.98.3:40663/ using document root <none> and password file <none>
I20260619 12:30:53.098100 10982 fs_manager.cc:359] Metadata directory not provided
I20260619 12:30:53.098214 10982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:30:53.098487 10982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:30:53.100766 10982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "668eff0798274f0d95cfe11cd7441883"
format_stamp: "Formatted at 2026-06-19 12:30:53 on dist-test-slave-fg3h"
server_key: "ea3af9e457dc0d053b2bcd45fe5ff48c"
server_key_iv: "12e05f57023088f0a4d237aaa2dd2ef9"
server_key_version: "encryptionkey@0"
I20260619 12:30:53.101438 10982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "668eff0798274f0d95cfe11cd7441883"
format_stamp: "Formatted at 2026-06-19 12:30:53 on dist-test-slave-fg3h"
server_key: "ea3af9e457dc0d053b2bcd45fe5ff48c"
server_key_iv: "12e05f57023088f0a4d237aaa2dd2ef9"
server_key_version: "encryptionkey@0"
I20260619 12:30:53.105633 10982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:30:53.108355 10997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:53.110008 10982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:30:53.110106 10982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "668eff0798274f0d95cfe11cd7441883"
format_stamp: "Formatted at 2026-06-19 12:30:53 on dist-test-slave-fg3h"
server_key: "ea3af9e457dc0d053b2bcd45fe5ff48c"
server_key_iv: "12e05f57023088f0a4d237aaa2dd2ef9"
server_key_version: "encryptionkey@0"
I20260619 12:30:53.110258 10982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:30:53.128556 10982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:30:53.131858 10982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:30:53.132077 10982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:30:53.132731 10982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:30:53.133647 10982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:30:53.133694 10982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:53.133733 10982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:30:53.133749 10982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:30:53.143370 10982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42453
I20260619 12:30:53.143391 11110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42453 every 8 connection(s)
I20260619 12:30:53.144268 10982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:30:53.149900 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 10982
I20260619 12:30:53.150035 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:30:53.150375 392 external_mini_cluster.cc:1428] Setting key c010d3ce7df6272f1101e76fd475dea6
Jun 19 12:30:53 dist-test-slave-fg3h krb5kdc[10607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:30:53.159403 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:53.146220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32935 (local address 127.0.98.62:33231)
0619 12:30:53.146381 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:53.146385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:53.147238 (+ 853us) server_negotiation.cc:407] Connection header received
0619 12:30:53.148586 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:53.148590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:53.148648 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:53.148771 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:53.150409 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.151082 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.152788 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.153016 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.153213 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:53.155720 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:53.155723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:53.155725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:53.155751 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:53.157228 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:53.157760 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:53.157763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:53.157765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:53.157860 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:53.158233 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:53.158236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:53.158237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:53.158469 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:53.158551 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:53.159155 (+ 604us) server_negotiation.cc:299] Negotiation successful
0619 12:30:53.159271 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 12:30:53.160360 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:53.146597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33231 (local address 127.0.98.3:32935)
0619 12:30:53.147066 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:53.147109 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:53.148349 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:53.148946 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:53.148953 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:53.149388 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:30:53.150210 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:53.150232 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.151227 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.151230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:53.152651 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:53.152661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.153219 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.153222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:53.153368 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:53.153962 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:30:53.153981 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:30:53.155559 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:30:53.157335 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:53.157340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:53.157352 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:53.157649 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:53.157981 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:30:53.157984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:30:53.157985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:30:53.158129 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:30:53.158571 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:30:53.158577 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:30:53.158811 (+ 234us) client_negotiation.cc:769] Sending connection context
0619 12:30:53.158978 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 12:30:53.159355 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":107,"threads_started":1}
I20260619 12:30:53.161767 11111 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33231
I20260619 12:30:53.161996 11111 heartbeater.cc:464] Registering TS with master...
I20260619 12:30:53.162446 11111 heartbeater.cc:511] Master 127.0.98.62:33231 requested a full tablet report, sending...
I20260619 12:30:53.163473 10652 ts_manager.cc:194] Registered new tserver with Master: 668eff0798274f0d95cfe11cd7441883 (127.0.98.3:42453)
I20260619 12:30:53.164175 10652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32935
I20260619 12:30:53.164584 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:30:53.174774 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:53.166659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44266 (local address 127.0.98.62:33231)
0619 12:30:53.166783 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:53.166787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:53.166902 (+ 115us) server_negotiation.cc:407] Connection header received
0619 12:30:53.167013 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:53.167015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:53.167050 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:53.167120 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:30:53.168067 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.168574 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.169544 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.169755 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.169846 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:53.171425 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:30:53.171428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:30:53.171430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:30:53.171452 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:30:53.172919 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:53.173386 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:53.173389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:53.173390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:53.173461 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:30:53.173820 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:30:53.173828 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:30:53.173831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:30:53.174081 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:30:53.174191 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:53.174457 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 12:30:53.174613 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 12:30:53.177740 10652 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:30:53.180727 10652 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:30:53.195881 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:53.189983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45867 (local address 127.0.0.1:36786)
0619 12:30:53.190564 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:53.190579 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:53.190710 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:53.191371 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:53.191377 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:53.191402 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:53.191544 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:53.191551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.193054 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.193057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:53.194068 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:53.194078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.195197 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.195201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:53.195310 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:53.195360 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 12:30:53.195526 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 12:30:53.195671 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":137,"thread_start_us":90,"threads_started":1}
I20260619 12:30:53.196031 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:53.190444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36786 (local address 127.0.98.2:45867)
0619 12:30:53.190832 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:53.190846 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:53.190871 (+ 25us) server_negotiation.cc:407] Connection header received
0619 12:30:53.190953 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:53.190958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:53.191201 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:53.191380 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:53.191685 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.192894 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.194202 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.195042 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.195169 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:53.195316 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:53.195793 (+ 477us) server_negotiation.cc:299] Negotiation successful
0619 12:30:53.195900 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":88,"threads_started":1}
I20260619 12:30:53.195881 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:53.190297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41335 (local address 127.0.0.1:51464)
0619 12:30:53.190893 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:53.190904 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:53.190969 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:53.191735 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:53.191738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:53.191756 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:53.191842 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:53.191857 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.192801 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.192807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:53.194456 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:53.194466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.195514 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.195517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:53.195626 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:53.195647 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 12:30:53.195687 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 12:30:53.195738 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":144,"thread_start_us":83,"threads_started":1}
I20260619 12:30:53.196753 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:53.189715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48862 (local address 127.0.98.3:42453)
0619 12:30:53.190994 (+ 1279us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:53.191006 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:53.191024 (+ 18us) server_negotiation.cc:407] Connection header received
0619 12:30:53.191091 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:53.191095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:53.191237 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:53.191370 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:53.191823 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.192945 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.194690 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.195312 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.195445 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:53.196452 (+ 1007us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:53.196505 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 12:30:53.196609 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1182,"thread_start_us":178,"threads_started":1}
I20260619 12:30:53.197609 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:53.190606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51464 (local address 127.0.98.1:41335)
0619 12:30:53.191230 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:53.191247 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:53.191271 (+ 24us) server_negotiation.cc:407] Connection header received
0619 12:30:53.191374 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:53.191379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:53.191551 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:53.191765 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:53.191952 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.192679 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.194594 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.195388 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.196720 (+ 1332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:53.197317 (+ 597us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:53.197366 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 12:30:53.197462 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":83,"threads_started":1}
I20260619 12:30:53.199626 10911 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0c32eb14bf48f9a75e8db555b0ec10 (DEFAULT_TABLE table=test-table [id=9fd7636cc87c4adfa48a33a6105468f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:53.200368 10777 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0c32eb14bf48f9a75e8db555b0ec10 (DEFAULT_TABLE table=test-table [id=9fd7636cc87c4adfa48a33a6105468f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:53.200806 10911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0c32eb14bf48f9a75e8db555b0ec10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:53.201371 10777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0c32eb14bf48f9a75e8db555b0ec10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:53.196092 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:53.189554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42453 (local address 127.0.0.1:48862)
0619 12:30:53.189938 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:53.190002 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:53.190147 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:53.191550 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:53.191552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:53.191568 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:53.191692 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:53.191698 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.193073 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.193076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:53.194563 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:53.194571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.195442 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.195449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:53.195547 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:53.195570 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:30:53.195631 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 12:30:53.195709 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":143,"thread_start_us":105,"threads_started":1}
I20260619 12:30:53.207940 11045 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0c32eb14bf48f9a75e8db555b0ec10 (DEFAULT_TABLE table=test-table [id=9fd7636cc87c4adfa48a33a6105468f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:30:53.208449 11128 tablet_bootstrap.cc:492] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84: Bootstrap starting.
I20260619 12:30:53.209029 11045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0c32eb14bf48f9a75e8db555b0ec10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:30:53.210487 11129 tablet_bootstrap.cc:492] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f: Bootstrap starting.
I20260619 12:30:53.211665 11128 tablet_bootstrap.cc:654] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:53.212525 11128 log.cc:824] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:53.212946 11129 tablet_bootstrap.cc:654] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:53.213801 11129 log.cc:824] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:53.214829 11128 tablet_bootstrap.cc:492] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84: No bootstrap required, opened a new log
I20260619 12:30:53.215039 11128 ts_tablet_manager.cc:1397] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 12:30:53.215997 11129 tablet_bootstrap.cc:492] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f: No bootstrap required, opened a new log
I20260619 12:30:53.216176 11129 ts_tablet_manager.cc:1397] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260619 12:30:53.216392 11132 tablet_bootstrap.cc:492] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883: Bootstrap starting.
I20260619 12:30:53.219262 11132 tablet_bootstrap.cc:654] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883: Neither blocks nor log segments found. Creating new log.
I20260619 12:30:53.219684 11128 raft_consensus.cc:348] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668eff0798274f0d95cfe11cd7441883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42453 } } peers { permanent_uuid: "2414cbf8ce09456b88972bce9ce56f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41335 } } peers { permanent_uuid: "c3db72d6eeed4db7b130b24d3361160f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } }
I20260619 12:30:53.219975 11128 raft_consensus.cc:374] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:53.219976 11132 log.cc:824] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883: Log is configured to *not* fsync() on all Append() calls
I20260619 12:30:53.220037 11128 raft_consensus.cc:729] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2414cbf8ce09456b88972bce9ce56f84, State: Initialized, Role: FOLLOWER
I20260619 12:30:53.220268 11129 raft_consensus.cc:348] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668eff0798274f0d95cfe11cd7441883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42453 } } peers { permanent_uuid: "2414cbf8ce09456b88972bce9ce56f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41335 } } peers { permanent_uuid: "c3db72d6eeed4db7b130b24d3361160f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } }
I20260619 12:30:53.220533 11129 raft_consensus.cc:374] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:53.220485 11128 consensus_queue.cc:260] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668eff0798274f0d95cfe11cd7441883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42453 } } peers { permanent_uuid: "2414cbf8ce09456b88972bce9ce56f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41335 } } peers { permanent_uuid: "c3db72d6eeed4db7b130b24d3361160f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } }
I20260619 12:30:53.220585 11129 raft_consensus.cc:729] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3db72d6eeed4db7b130b24d3361160f, State: Initialized, Role: FOLLOWER
I20260619 12:30:53.220992 11129 consensus_queue.cc:260] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668eff0798274f0d95cfe11cd7441883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42453 } } peers { permanent_uuid: "2414cbf8ce09456b88972bce9ce56f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41335 } } peers { permanent_uuid: "c3db72d6eeed4db7b130b24d3361160f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } }
I20260619 12:30:53.221522 11128 ts_tablet_manager.cc:1428] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260619 12:30:53.221554 10843 heartbeater.cc:503] Master 127.0.98.62:33231 was elected leader, sending a full tablet report...
I20260619 12:30:53.221880 11129 ts_tablet_manager.cc:1428] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:30:53.221958 11132 tablet_bootstrap.cc:492] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883: No bootstrap required, opened a new log
I20260619 12:30:53.222122 11132 ts_tablet_manager.cc:1397] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:30:53.222186 10977 heartbeater.cc:503] Master 127.0.98.62:33231 was elected leader, sending a full tablet report...
I20260619 12:30:53.225996 11132 raft_consensus.cc:348] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668eff0798274f0d95cfe11cd7441883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42453 } } peers { permanent_uuid: "2414cbf8ce09456b88972bce9ce56f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41335 } } peers { permanent_uuid: "c3db72d6eeed4db7b130b24d3361160f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } }
I20260619 12:30:53.226253 11132 raft_consensus.cc:374] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:30:53.226290 11132 raft_consensus.cc:729] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668eff0798274f0d95cfe11cd7441883, State: Initialized, Role: FOLLOWER
I20260619 12:30:53.226729 11132 consensus_queue.cc:260] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668eff0798274f0d95cfe11cd7441883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42453 } } peers { permanent_uuid: "2414cbf8ce09456b88972bce9ce56f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41335 } } peers { permanent_uuid: "c3db72d6eeed4db7b130b24d3361160f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } }
I20260619 12:30:53.227409 11111 heartbeater.cc:503] Master 127.0.98.62:33231 was elected leader, sending a full tablet report...
I20260619 12:30:53.227675 11132 ts_tablet_manager.cc:1428] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260619 12:30:53.395283 11112 tablet.cc:2369] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:30:53.411947 10978 tablet.cc:2369] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:30:53.426136 10844 tablet.cc:2369] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:30:53.489406 11134 raft_consensus.cc:482] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:30:53.489679 11134 raft_consensus.cc:504] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668eff0798274f0d95cfe11cd7441883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42453 } } peers { permanent_uuid: "2414cbf8ce09456b88972bce9ce56f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41335 } } peers { permanent_uuid: "c3db72d6eeed4db7b130b24d3361160f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } }
I20260619 12:30:53.490865 11134 leader_election.cc:288] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 668eff0798274f0d95cfe11cd7441883 (127.0.98.3:42453), c3db72d6eeed4db7b130b24d3361160f (127.0.98.2:45867)
I20260619 12:30:53.496194 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:53.491328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59767 (local address 127.0.98.2:45867)
0619 12:30:53.491455 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:53.491460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:53.491724 (+ 264us) server_negotiation.cc:407] Connection header received
0619 12:30:53.491807 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:53.491810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:53.491869 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:53.491936 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:53.492282 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.493063 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.494904 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.495523 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.495643 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:53.495712 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:53.495992 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 12:30:53.496062 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:30:53.496194 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:53.491483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58347 (local address 127.0.98.3:42453)
0619 12:30:53.491641 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:53.491645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:53.491659 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:30:53.491737 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:53.491741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:53.491799 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:53.491877 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:30:53.492282 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.493070 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.494904 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.495523 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.495654 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:53.495709 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:53.495992 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 12:30:53.496062 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:30:53.496269 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:53.491087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42453 (local address 127.0.98.1:58347)
0619 12:30:53.491490 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:53.491505 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:53.491623 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:53.491945 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:53.491949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:53.491970 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:53.492146 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:53.492153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.493224 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.493227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:53.494749 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:53.494761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.495664 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.495668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:53.495811 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:53.495872 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 12:30:53.495952 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 12:30:53.496081 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":233,"thread_start_us":157,"threads_started":1}
I20260619 12:30:53.496461 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:53.491096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45867 (local address 127.0.98.1:59767)
0619 12:30:53.491613 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:53.491624 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:30:53.491712 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:30:53.491963 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:30:53.491966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:30:53.491976 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:30:53.492146 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:53.492153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.493224 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.493227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:53.494749 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:30:53.494761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.495664 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.495668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:30:53.495814 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:53.495872 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 12:30:53.495952 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 12:30:53.496083 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":166,"thread_start_us":138,"threads_started":1}
I20260619 12:30:53.496985 11065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0c32eb14bf48f9a75e8db555b0ec10" candidate_uuid: "2414cbf8ce09456b88972bce9ce56f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668eff0798274f0d95cfe11cd7441883" is_pre_election: true
I20260619 12:30:53.497149 10931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0c32eb14bf48f9a75e8db555b0ec10" candidate_uuid: "2414cbf8ce09456b88972bce9ce56f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3db72d6eeed4db7b130b24d3361160f" is_pre_election: true
I20260619 12:30:53.497400 11065 raft_consensus.cc:2454] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2414cbf8ce09456b88972bce9ce56f84 in term 0.
I20260619 12:30:53.497449 10931 raft_consensus.cc:2454] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2414cbf8ce09456b88972bce9ce56f84 in term 0.
I20260619 12:30:53.497969 10732 leader_election.cc:302] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2414cbf8ce09456b88972bce9ce56f84, 668eff0798274f0d95cfe11cd7441883, c3db72d6eeed4db7b130b24d3361160f; no voters:
I20260619 12:30:53.498265 11134 raft_consensus.cc:2781] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:30:53.498343 11134 raft_consensus.cc:482] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:30:53.498375 11134 raft_consensus.cc:3037] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:53.499447 11134 raft_consensus.cc:504] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668eff0798274f0d95cfe11cd7441883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42453 } } peers { permanent_uuid: "2414cbf8ce09456b88972bce9ce56f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41335 } } peers { permanent_uuid: "c3db72d6eeed4db7b130b24d3361160f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } }
I20260619 12:30:53.499862 11134 leader_election.cc:288] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 [CANDIDATE]: Term 1 election: Requested vote from peers 668eff0798274f0d95cfe11cd7441883 (127.0.98.3:42453), c3db72d6eeed4db7b130b24d3361160f (127.0.98.2:45867)
I20260619 12:30:53.500248 11065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0c32eb14bf48f9a75e8db555b0ec10" candidate_uuid: "2414cbf8ce09456b88972bce9ce56f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "668eff0798274f0d95cfe11cd7441883"
I20260619 12:30:53.500285 10931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0c32eb14bf48f9a75e8db555b0ec10" candidate_uuid: "2414cbf8ce09456b88972bce9ce56f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3db72d6eeed4db7b130b24d3361160f"
I20260619 12:30:53.500351 11065 raft_consensus.cc:3037] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:53.500432 10931 raft_consensus.cc:3037] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:30:53.501470 10931 raft_consensus.cc:2454] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2414cbf8ce09456b88972bce9ce56f84 in term 1.
I20260619 12:30:53.501487 11065 raft_consensus.cc:2454] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2414cbf8ce09456b88972bce9ce56f84 in term 1.
I20260619 12:30:53.501806 10732 leader_election.cc:302] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2414cbf8ce09456b88972bce9ce56f84, c3db72d6eeed4db7b130b24d3361160f; no voters:
I20260619 12:30:53.501968 11134 raft_consensus.cc:2781] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:30:53.502188 11134 raft_consensus.cc:686] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 [term 1 LEADER]: Becoming Leader. State: Replica: 2414cbf8ce09456b88972bce9ce56f84, State: Running, Role: LEADER
I20260619 12:30:53.502521 11134 consensus_queue.cc:237] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668eff0798274f0d95cfe11cd7441883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42453 } } peers { permanent_uuid: "2414cbf8ce09456b88972bce9ce56f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41335 } } peers { permanent_uuid: "c3db72d6eeed4db7b130b24d3361160f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } }
I20260619 12:30:53.505610 10652 catalog_manager.cc:5495] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2414cbf8ce09456b88972bce9ce56f84 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2414cbf8ce09456b88972bce9ce56f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668eff0798274f0d95cfe11cd7441883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2414cbf8ce09456b88972bce9ce56f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3db72d6eeed4db7b130b24d3361160f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 } health_report { overall_health: UNKNOWN } } }
I20260619 12:30:53.571679 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:53.567882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51476 (local address 127.0.98.1:41335)
0619 12:30:53.568018 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 12:30:53.568024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:30:53.568041 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:30:53.568138 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:30:53.568140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:30:53.568195 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:30:53.568309 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:30:53.568578 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.569144 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.570086 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:30:53.570307 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:30:53.570424 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:30:53.570759 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:30:53.571275 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:30:53.571433 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 12:30:53.571468 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 12:30:53.571527 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:30:53.577718 11065 raft_consensus.cc:1261] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883 [term 1 FOLLOWER]: Refusing update from remote peer 2414cbf8ce09456b88972bce9ce56f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:53.577718 10931 raft_consensus.cc:1261] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f [term 1 FOLLOWER]: Refusing update from remote peer 2414cbf8ce09456b88972bce9ce56f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:30:53.578415 11139 consensus_queue.cc:1035] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3db72d6eeed4db7b130b24d3361160f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:53.578642 11134 consensus_queue.cc:1035] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 2414cbf8ce09456b88972bce9ce56f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668eff0798274f0d95cfe11cd7441883" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:30:53.594290 10652 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:53.594480 10652 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:30:53.596932 10652 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aad94b5232d843fab881b6bbe9b728be: Sending DeleteTablet for 3 replicas of tablet 0c0c32eb14bf48f9a75e8db555b0ec10
I20260619 12:30:53.597779 10777 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0c32eb14bf48f9a75e8db555b0ec10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51464
I20260619 12:30:53.597791 11045 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0c32eb14bf48f9a75e8db555b0ec10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48862
I20260619 12:30:53.598301 11152 tablet_replica.cc:330] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883: stopping tablet replica
I20260619 12:30:53.598408 10911 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0c32eb14bf48f9a75e8db555b0ec10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:30:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36786
I20260619 12:30:53.598495 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 10714
I20260619 12:30:53.598690 11152 raft_consensus.cc:2229] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:53.599375 11152 raft_consensus.cc:2258] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:53.600296 11153 tablet_replica.cc:330] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f: stopping tablet replica
I20260619 12:30:53.600605 11153 raft_consensus.cc:2229] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:30:53.600932 11153 raft_consensus.cc:2258] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:30:53.601116 11152 ts_tablet_manager.cc:1905] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:53.602196 11153 ts_tablet_manager.cc:1905] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:30:53.604256 11153 ts_tablet_manager.cc:1918] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:30:53.604382 11153 log.cc:1196] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/0c0c32eb14bf48f9a75e8db555b0ec10
I20260619 12:30:53.604542 11152 ts_tablet_manager.cc:1918] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:30:53.604660 11152 log.cc:1196] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/0c0c32eb14bf48f9a75e8db555b0ec10
I20260619 12:30:53.604717 11153 ts_tablet_manager.cc:1939] T 0c0c32eb14bf48f9a75e8db555b0ec10 P c3db72d6eeed4db7b130b24d3361160f: Deleting consensus metadata
I20260619 12:30:53.604988 11152 ts_tablet_manager.cc:1939] T 0c0c32eb14bf48f9a75e8db555b0ec10 P 668eff0798274f0d95cfe11cd7441883: Deleting consensus metadata
I20260619 12:30:53.605890 10638 catalog_manager.cc:4841] TS c3db72d6eeed4db7b130b24d3361160f (127.0.98.2:45867): tablet 0c0c32eb14bf48f9a75e8db555b0ec10 (table test-table [id=9fd7636cc87c4adfa48a33a6105468f8]) successfully deleted
I20260619 12:30:53.606153 10636 catalog_manager.cc:4841] TS 668eff0798274f0d95cfe11cd7441883 (127.0.98.3:42453): tablet 0c0c32eb14bf48f9a75e8db555b0ec10 (table test-table [id=9fd7636cc87c4adfa48a33a6105468f8]) successfully deleted
W20260619 12:30:53.606793 10636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41335 (error 108)
I20260619 12:30:53.607087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 10848
I20260619 12:30:53.608180 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:30:53.607828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41335 (local address 127.0.0.1:51492)
0619 12:30:53.607991 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 12:30:53.608076 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 12:30:53.608366 10636 catalog_manager.cc:4584] TS 2414cbf8ce09456b88972bce9ce56f84 (127.0.98.1:41335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c0c32eb14bf48f9a75e8db555b0ec10: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41335: connect: Connection refused (error 111)
I20260619 12:30:53.613412 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 10982
I20260619 12:30:53.619282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 10620
2026-06-19T12:30:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2609 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:30:53 dist-test-slave-fg3h krb5kdc[11157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:30:53 dist-test-slave-fg3h krb5kdc[11157](info): set up 2 sockets
Jun 19 12:30:53 dist-test-slave-fg3h krb5kdc[11157](info): commencing operation
krb5kdc: starting...
W20260619 12:30:54.686110 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:30:54 dist-test-slave-fg3h krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:30:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:30:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 12:30:54.863953 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskf39FOk/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/postgres -l logfile start
2026-06-19 12:30:57.584 UTC [11187] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 12:30:57.584 UTC [11187] LOG: listening on IPv4 address "127.0.98.20", port 36787
2026-06-19 12:30:57.588 UTC [11187] LOG: listening on Unix socket "/tmp/.s.PGSQL.36787"
2026-06-19 12:30:57.611 UTC [11190] LOG: database system was shut down at 2026-06-19 12:30:55 UTC
2026-06-19 12:30:57.615 UTC [11187] LOG: database system is ready to accept connections
I20260619 12:30:58.569921 392 mini_postgres.cc:96] Postgres bound to 36787
2026-06-19 12:30:58.576 UTC [11198] FATAL: database "slave" does not exist
127.0.98.20:36787 - accepting connections
I20260619 12:30:58.576911 392 mini_ranger.cc:161] Starting Ranger...
I20260619 12:30:58.591735 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 12:30:59.146067 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 12:30:59.146162 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ranger-admin
2026-06-19 12:30:59,443 [I] DB FLAVOR :POSTGRES
2026-06-19 12:30:59,444 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 12:30:59,444 [I] Checking connection..
2026-06-19 12:30:59,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 12:30:59,699 [I] Checking connection passed.
2026-06-19 12:30:59,699 [I] --------- Verifying version history table ---------
2026-06-19 12:30:59,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 12:30:59,956 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 12:30:59,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 12:31:00,228 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 12:31:00,228 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 12:31:00,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 12:31:00.453 UTC [11262] WARNING: there is no transaction in progress
2026-06-19 12:31:00,460 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 12:31:00,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 12:31:00,699 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 12:31:00,699 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 12:31:00,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 12:31:00,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 12:31:01,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fg3h.c.gcp-upstream.internal','N') ;"
2026-06-19 12:31:01,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 12:31:01,661 [I] Table x_portal_user does not exist in database ranger
2026-06-19 12:31:01,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 12:31:01,889 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 12:31:01,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 12:31:02,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 12:31:02,408 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 12:31:02,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 12:31:03.376 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 12:31:03.393 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 12:31:03.409 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 12:31:03.427 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 12:31:03.444 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 12:31:03.465 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 12:31:03.523 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 12:31:03.531 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 12:31:03.547 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 12:31:03.558 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 12:31:03.571 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 12:31:03.585 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 12:31:03.596 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 12:31:03.603 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 12:31:03.611 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 12:31:03.619 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 12:31:04,304 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 12:31:04,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 12:31:04,528 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 12:31:04,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 12:31:04,762 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 12:31:04,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 12:31:05,005 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 12:31:05,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 12:31:05,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 12:31:05,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fg3h.c.gcp-upstream.internal';"
2026-06-19 12:31:05,727 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 12:31:05,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 12:31:05,986 [I] DB_PATCHES have already been applied
I20260619 12:31:05.992242 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskf39FOk/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 12:31:05.992311 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 12:31:05.995899 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:37359
Jun 19, 2026 12:31:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 12:31:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 12:31:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 12:31:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 12:31:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 12:31:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 12:31:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 12:31:07.173990 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.597s user 0.000s sys 0.010s
Jun 19, 2026 12:31:26 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 12:31:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 12:31:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 12:31:26 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 12:31:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 12:31:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 12:31:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 12:31:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 12:31:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 12:31:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 12:31:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 12:31:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 12:31:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 12:31:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 12:31:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 12:31:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 12:31:26 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 12:31:27 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 12:31:28.419723 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 12:31:28 dist-test-slave-fg3h krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872288, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 12:31:28.503005 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 12:31:28.512743 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 12:31:29.390466 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 12:31:29.390568 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ranger-kms
I20260619 12:31:29.390669 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:36787
2026-06-19 12:31:29,579 [I] DB FLAVOR :POSTGRES
2026-06-19 12:31:29,580 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 12:31:29,580 [I] Checking connection
2026-06-19 12:31:29,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 12:31:29,845 [I] connection success
2026-06-19 12:31:29,845 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 12:31:29,845 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 12:31:29,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 12:31:30,103 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 12:31:30,103 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 12:31:30,103 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 12:31:30,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36787/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 12:31:30,370 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 12:31:30.824164 392 mini_ranger_kms.cc:310] Created kms service
I20260619 12:31:31.032697 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 12:31:31.079912 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 12:31:31 dist-test-slave-fg3h krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872288, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:31:31.465709 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 12:31:31.465862 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 12:31:31.465886 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 12:31:31.470562 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11550
I20260619 12:31:31.470721 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:34619
Jun 19, 2026 12:31:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 12:31:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 12:31:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 12:31:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 12:31:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 12:31:32 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 12:31:33.717813 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.215s user 0.007s sys 0.162s
I20260619 12:31:33.717922 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 12:31:33.718048 392 mini_ranger_kms.cc:401] 127.0.98.20:34619/kms/v1/keys
Jun 19 12:31:33 dist-test-slave-fg3h krb5kdc[11157](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781872293, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 12:31:34 dist-test-slave-fg3h krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872293, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 12:31:34 dist-test-slave-fg3h krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872293, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 12:31:36 dist-test-slave-fg3h krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:31:36.804096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45979
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45979
--ranger_config_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34619/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:31:36.917745 11601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:31:36.918058 11601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:31:36.918128 11601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:31:36.922180 11601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:31:36.922281 11601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:31:36.922317 11601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:31:36.922370 11601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:31:36.922422 11601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:31:36.928069 11601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:34619/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45979
--ranger_config_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:31:36.929327 11601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:31:36.930375 11601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:31:36.939254 11609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:31:36.939357 11607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:31:36.939543 11601 server_base.cc:1034] running on GCE node
W20260619 12:31:36.939349 11606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:31:36.940208 11601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:31:36.941336 11601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:31:36.942541 11601 hybrid_clock.cc:648] HybridClock initialized: now 1781872296942527 us; error 29 us; skew 500 ppm
Jun 19 12:31:36 dist-test-slave-fg3h krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:31:36.945363 11601 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:31:36.946317 11601 webserver.cc:460] Webserver started at http://127.0.98.62:37213/ using document root <none> and password file <none>
I20260619 12:31:36.946816 11601 fs_manager.cc:359] Metadata directory not provided
I20260619 12:31:36.946868 11601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:31:36.947034 11601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:31:36 dist-test-slave-fg3h krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:31:37.010388 11601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "190d743b4f1147da901cfda7725cea1e"
format_stamp: "Formatted at 2026-06-19 12:31:37 on dist-test-slave-fg3h"
server_key: "e20d09d27d00e254edc317fa106353a2"
server_key_iv: "e0cf8f7e769e33360ea47396ad764066"
server_key_version: "kuduclusterkey@0"
I20260619 12:31:37.011161 11601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "190d743b4f1147da901cfda7725cea1e"
format_stamp: "Formatted at 2026-06-19 12:31:37 on dist-test-slave-fg3h"
server_key: "e20d09d27d00e254edc317fa106353a2"
server_key_iv: "e0cf8f7e769e33360ea47396ad764066"
server_key_version: "kuduclusterkey@0"
I20260619 12:31:37.015321 11601 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:31:37.032969 11616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:31:37.034778 11601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:31:37.034948 11601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "190d743b4f1147da901cfda7725cea1e"
format_stamp: "Formatted at 2026-06-19 12:31:37 on dist-test-slave-fg3h"
server_key: "e20d09d27d00e254edc317fa106353a2"
server_key_iv: "e0cf8f7e769e33360ea47396ad764066"
server_key_version: "kuduclusterkey@0"
I20260619 12:31:37.035104 11601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:31:37.070291 11601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:31:37.073441 11601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:31:37.073617 11601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:31:37.081334 11601 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45979
I20260619 12:31:37.081360 11668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45979 every 8 connection(s)
I20260619 12:31:37.082343 11601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:31:37.085228 11669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:31:37.090801 11669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e: Bootstrap starting.
I20260619 12:31:37.092549 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 11601
I20260619 12:31:37.092685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:31:37.092986 392 external_mini_cluster.cc:1428] Setting key c82723f8572ac87ec7e93dd03a497988
I20260619 12:31:37.093612 11669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e: Neither blocks nor log segments found. Creating new log.
I20260619 12:31:37.094523 11669 log.cc:824] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e: Log is configured to *not* fsync() on all Append() calls
I20260619 12:31:37.096395 11669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e: No bootstrap required, opened a new log
I20260619 12:31:37.099846 11669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190d743b4f1147da901cfda7725cea1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45979 } }
I20260619 12:31:37.100163 11669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:31:37.100245 11669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190d743b4f1147da901cfda7725cea1e, State: Initialized, Role: FOLLOWER
I20260619 12:31:37.100725 11669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190d743b4f1147da901cfda7725cea1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45979 } }
I20260619 12:31:37.100894 11669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:31:37.100977 11669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:31:37.101076 11669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e [term 0 FOLLOWER]: Advancing to term 1
Jun 19 12:31:37 dist-test-slave-fg3h krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:31:37.102276 11669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190d743b4f1147da901cfda7725cea1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45979 } }
I20260619 12:31:37.102635 11669 leader_election.cc:302] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 190d743b4f1147da901cfda7725cea1e; no voters:
I20260619 12:31:37.103008 11669 leader_election.cc:288] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:31:37.103121 11674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:31:37.103391 11674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e [term 1 LEADER]: Becoming Leader. State: Replica: 190d743b4f1147da901cfda7725cea1e, State: Running, Role: LEADER
I20260619 12:31:37.103790 11674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190d743b4f1147da901cfda7725cea1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45979 } }
I20260619 12:31:37.104383 11669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:31:37.105734 11675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "190d743b4f1147da901cfda7725cea1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190d743b4f1147da901cfda7725cea1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45979 } } }
I20260619 12:31:37.105886 11675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e [sys.catalog]: This master's current role is: LEADER
I20260619 12:31:37.105732 11676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 190d743b4f1147da901cfda7725cea1e. Latest consensus state: current_term: 1 leader_uuid: "190d743b4f1147da901cfda7725cea1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190d743b4f1147da901cfda7725cea1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45979 } } }
I20260619 12:31:37.106186 11676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e [sys.catalog]: This master's current role is: LEADER
I20260619 12:31:37.106583 11685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:31:37.106675 11669 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 12:31:37.111469 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:37.094176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37078 (local address 127.0.98.62:45979)
0619 12:31:37.094607 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 12:31:37.094622 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:31:37.094679 (+ 57us) server_negotiation.cc:407] Connection header received
0619 12:31:37.095393 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:31:37.095412 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:31:37.095738 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:31:37.096094 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:31:37.096903 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:37.097943 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:37.098816 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:37.099300 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:37.099463 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:31:37.102552 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:31:37.102558 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:31:37.102572 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:31:37.102600 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:31:37.104611 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:31:37.105900 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:31:37.105907 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:31:37.105913 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:31:37.106061 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:31:37.106393 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:31:37.106397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:31:37.106399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:31:37.106937 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:31:37.107143 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 12:31:37.107419 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 12:31:37.107705 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":128,"threads_started":1}
I20260619 12:31:37.114575 11685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:31:37.121713 11685 catalog_manager.cc:1347] Generated new cluster ID: ed4040fd02764e4b8f0f5d4caa2d16fe
I20260619 12:31:37.121794 11685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:31:37.134919 11685 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:31:37.136327 11685 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:31:37.144544 11685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e: Generated new TSK 0
I20260619 12:31:37.145432 11685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:31:37.827986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34619/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45979
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:31:37.981619 11714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:31:37.981988 11714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:31:37.982050 11714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:31:37.988725 11714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:31:37.988817 11714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:31:37.988921 11714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:31:37.997404 11714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:34619/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:31:37.998842 11714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:31:38.000252 11714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:31:38.012871 11721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:31:38.012995 11722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:31:38.012873 11724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:31:38.014117 11714 server_base.cc:1034] running on GCE node
I20260619 12:31:38.014639 11714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:31:38.015497 11714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:31:38.016697 11714 hybrid_clock.cc:648] HybridClock initialized: now 1781872298016657 us; error 58 us; skew 500 ppm
Jun 19 12:31:38 dist-test-slave-fg3h krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:31:38.020488 11714 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:31:38.021760 11714 webserver.cc:460] Webserver started at http://127.0.98.1:38631/ using document root <none> and password file <none>
I20260619 12:31:38.022540 11714 fs_manager.cc:359] Metadata directory not provided
I20260619 12:31:38.022644 11714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:31:38.023005 11714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:31:38 dist-test-slave-fg3h krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:31:38.040565 11714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "9f84da732c7c40559d26bcbbdb099854"
format_stamp: "Formatted at 2026-06-19 12:31:38 on dist-test-slave-fg3h"
server_key: "52e1fed4d6081761a9b7af3b256dbf38"
server_key_iv: "42ec8c1e59ea6c592008860933f777dd"
server_key_version: "kuduclusterkey@0"
I20260619 12:31:38.041328 11714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9f84da732c7c40559d26bcbbdb099854"
format_stamp: "Formatted at 2026-06-19 12:31:38 on dist-test-slave-fg3h"
server_key: "52e1fed4d6081761a9b7af3b256dbf38"
server_key_iv: "42ec8c1e59ea6c592008860933f777dd"
server_key_version: "kuduclusterkey@0"
I20260619 12:31:38.056015 11714 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.003s sys 0.001s
I20260619 12:31:38.075961 11731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:31:38.081480 11714 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260619 12:31:38.081710 11714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "9f84da732c7c40559d26bcbbdb099854"
format_stamp: "Formatted at 2026-06-19 12:31:38 on dist-test-slave-fg3h"
server_key: "52e1fed4d6081761a9b7af3b256dbf38"
server_key_iv: "42ec8c1e59ea6c592008860933f777dd"
server_key_version: "kuduclusterkey@0"
I20260619 12:31:38.081948 11714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:31:38.117553 11714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:31:38.121448 11714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:31:38.121712 11714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:31:38.122511 11714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:31:38.123605 11714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:31:38.123719 11714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:31:38.123848 11714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:31:38.123934 11714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:31:38.137575 11714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45207
I20260619 12:31:38.138856 11714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:31:38.147985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 11714
I20260619 12:31:38.148116 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:31:38.148420 392 external_mini_cluster.cc:1428] Setting key 78cbd4fefc223d4b839d85110f479512
I20260619 12:31:38.140859 11846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45207 every 8 connection(s)
Jun 19 12:31:38 dist-test-slave-fg3h krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:31:38.162314 11850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:38.143408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45979 (local address 127.0.98.1:36669)
0619 12:31:38.145256 (+ 1848us) negotiation.cc:107] Waiting for socket to connect
0619 12:31:38.145314 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 12:31:38.146334 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:31:38.147315 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:31:38.147324 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:31:38.147724 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:31:38.148655 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:31:38.148676 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.151022 (+ 2346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.151026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:31:38.152436 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:31:38.152449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.153244 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.153248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:31:38.153444 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:31:38.154376 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:31:38.154398 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:31:38.156167 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:31:38.158593 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:31:38.158602 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:31:38.158621 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:31:38.158985 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:31:38.159335 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:31:38.159341 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:31:38.159346 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:31:38.159539 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:31:38.160059 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:31:38.160069 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:31:38.160462 (+ 393us) client_negotiation.cc:769] Sending connection context
0619 12:31:38.160689 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 12:31:38.161059 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1582,"thread_start_us":655,"threads_started":1}
I20260619 12:31:38.163084 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:38.143910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36669 (local address 127.0.98.62:45979)
0619 12:31:38.146784 (+ 2874us) server_negotiation.cc:206] Beginning negotiation
0619 12:31:38.146796 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:31:38.146811 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:31:38.146855 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:31:38.146858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:31:38.146907 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:31:38.146995 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:31:38.148829 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.150866 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.152580 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.152945 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.153047 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:31:38.156367 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:31:38.156371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:31:38.156374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:31:38.156401 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:31:38.158444 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:31:38.159105 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:31:38.159112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:31:38.159129 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:31:38.159231 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:31:38.159653 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:31:38.159660 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:31:38.159664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:31:38.159917 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:31:38.160006 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 12:31:38.162756 (+ 2750us) server_negotiation.cc:299] Negotiation successful
0619 12:31:38.162890 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2795,"thread_start_us":102,"threads_started":1}
I20260619 12:31:38.164548 11847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45979
I20260619 12:31:38.164808 11847 heartbeater.cc:464] Registering TS with master...
I20260619 12:31:38.165424 11847 heartbeater.cc:511] Master 127.0.98.62:45979 requested a full tablet report, sending...
I20260619 12:31:38.167634 11633 ts_manager.cc:194] Registered new tserver with Master: 9f84da732c7c40559d26bcbbdb099854 (127.0.98.1:45207)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:31:38.169332 11633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 19 12:31:38 dist-test-slave-fg3h krb5kdc[11157](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781872298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:31:38.201815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34619/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45979
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:31:38.379405 11853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:31:38.379884 11853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:31:38.380009 11853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:31:38.386442 11853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:31:38.386585 11853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:31:38.386744 11853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:31:38.394819 11853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:34619/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:31:38.396485 11853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:31:38.397966 11853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:31:38.411408 11859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:31:38.412858 11853 server_base.cc:1034] running on GCE node
W20260619 12:31:38.413555 11860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:31:38.419751 11862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:31:38.420469 11853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:31:38.421350 11853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:31:38.422681 11853 hybrid_clock.cc:648] HybridClock initialized: now 1781872298422608 us; error 93 us; skew 500 ppm
Jun 19 12:31:38 dist-test-slave-fg3h krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:31:38.426393 11853 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:31:38.427503 11853 webserver.cc:460] Webserver started at http://127.0.98.2:40371/ using document root <none> and password file <none>
I20260619 12:31:38.428205 11853 fs_manager.cc:359] Metadata directory not provided
I20260619 12:31:38.428275 11853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:31:38.428494 11853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:31:38 dist-test-slave-fg3h krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:31:38.448343 11853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "ca69e4e481d445219451fb703f7cd92b"
format_stamp: "Formatted at 2026-06-19 12:31:38 on dist-test-slave-fg3h"
server_key: "9d1ec3c158edd578c51b26c87b472ee5"
server_key_iv: "b5ed99c02bb7bf741fcb9b429a96afc6"
server_key_version: "kuduclusterkey@0"
I20260619 12:31:38.449265 11853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ca69e4e481d445219451fb703f7cd92b"
format_stamp: "Formatted at 2026-06-19 12:31:38 on dist-test-slave-fg3h"
server_key: "9d1ec3c158edd578c51b26c87b472ee5"
server_key_iv: "b5ed99c02bb7bf741fcb9b429a96afc6"
server_key_version: "kuduclusterkey@0"
I20260619 12:31:38.457326 11853 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.001s
I20260619 12:31:38.476155 11869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:31:38.478374 11853 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260619 12:31:38.478523 11853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "ca69e4e481d445219451fb703f7cd92b"
format_stamp: "Formatted at 2026-06-19 12:31:38 on dist-test-slave-fg3h"
server_key: "9d1ec3c158edd578c51b26c87b472ee5"
server_key_iv: "b5ed99c02bb7bf741fcb9b429a96afc6"
server_key_version: "kuduclusterkey@0"
I20260619 12:31:38.478674 11853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:31:38.508302 11853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:31:38.512001 11853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:31:38.512202 11853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:31:38.512867 11853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:31:38.513922 11853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:31:38.513993 11853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:31:38.514048 11853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:31:38.514120 11853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 12:31:38.526368 11853 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45735
I20260619 12:31:38.527587 11853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:31:38.534099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 11853
I20260619 12:31:38.534271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:31:38.534578 392 external_mini_cluster.cc:1428] Setting key b734e9eb72c7ff52ef310ce2516d04cf
I20260619 12:31:38.545989 11982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45735 every 8 connection(s)
Jun 19 12:31:38 dist-test-slave-fg3h krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:31:38.562501 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:38.530705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46621 (local address 127.0.98.62:45979)
0619 12:31:38.530860 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 12:31:38.530865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:31:38.547040 (+ 16175us) server_negotiation.cc:407] Connection header received
0619 12:31:38.548314 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:31:38.548318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:31:38.548378 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:31:38.548460 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:31:38.551381 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.552247 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.554062 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.554443 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.554521 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:31:38.557655 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:31:38.557659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:31:38.557662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:31:38.557691 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:31:38.559596 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:31:38.560260 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:31:38.560264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:31:38.560266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:31:38.560357 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:31:38.561066 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:31:38.561069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:31:38.561071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:31:38.561350 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:31:38.561441 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 12:31:38.562183 (+ 742us) server_negotiation.cc:299] Negotiation successful
0619 12:31:38.562327 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 12:31:38.564494 11985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:38.530887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45979 (local address 127.0.98.2:46621)
0619 12:31:38.546862 (+ 15975us) negotiation.cc:107] Waiting for socket to connect
0619 12:31:38.546928 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 12:31:38.548064 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:31:38.550083 (+ 2019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:31:38.550094 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:31:38.550486 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:31:38.551194 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:31:38.551214 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.552417 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.552420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:31:38.553921 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:31:38.553932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.554730 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.554734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:31:38.554935 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:31:38.555661 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:31:38.555683 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:31:38.557461 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:31:38.559740 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:31:38.559747 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:31:38.559765 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:31:38.560138 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:31:38.560768 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:31:38.560771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:31:38.560773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:31:38.560954 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:31:38.561512 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:31:38.561519 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:31:38.561871 (+ 352us) client_negotiation.cc:769] Sending connection context
0619 12:31:38.563038 (+ 1167us) client_negotiation.cc:240] Negotiation successful
0619 12:31:38.563402 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15657,"thread_start_us":182,"threads_started":1}
I20260619 12:31:38.566170 11983 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45979
I20260619 12:31:38.566421 11983 heartbeater.cc:464] Registering TS with master...
I20260619 12:31:38.566994 11983 heartbeater.cc:511] Master 127.0.98.62:45979 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:31:38.568183 11633 ts_manager.cc:194] Registered new tserver with Master: ca69e4e481d445219451fb703f7cd92b (127.0.98.2:45735)
I20260619 12:31:38.568962 11633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:31:38.597867 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34619/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45979
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:31:38 dist-test-slave-fg3h krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 12:31:38.760705 11988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:31:38.761207 11988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:31:38.761327 11988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:31:38.766433 11988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:31:38.766512 11988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:31:38.766625 11988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:31:38.774389 11988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:34619/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:31:38.776086 11988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:31:38.777479 11988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:31:38.789255 11996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:31:38.789989 11988 server_base.cc:1034] running on GCE node
W20260619 12:31:38.790266 11994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:31:38.791452 11993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:31:38.792064 11988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:31:38.792972 11988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:31:38.794162 11988 hybrid_clock.cc:648] HybridClock initialized: now 1781872298794134 us; error 43 us; skew 500 ppm
Jun 19 12:31:38 dist-test-slave-fg3h krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:31:38.798058 11988 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:31:38.799261 11988 webserver.cc:460] Webserver started at http://127.0.98.3:38725/ using document root <none> and password file <none>
I20260619 12:31:38.800037 11988 fs_manager.cc:359] Metadata directory not provided
I20260619 12:31:38.800197 11988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:31:38.800493 11988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:31:38 dist-test-slave-fg3h krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:31:38.817703 11988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "6e195552ad2b405d960744b5c002e84a"
format_stamp: "Formatted at 2026-06-19 12:31:38 on dist-test-slave-fg3h"
server_key: "514c10063d857aca3e4aea5745a24f22"
server_key_iv: "2e347a7e663a1ef0b928f8546d42bae8"
server_key_version: "kuduclusterkey@0"
I20260619 12:31:38.818830 11988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6e195552ad2b405d960744b5c002e84a"
format_stamp: "Formatted at 2026-06-19 12:31:38 on dist-test-slave-fg3h"
server_key: "514c10063d857aca3e4aea5745a24f22"
server_key_iv: "2e347a7e663a1ef0b928f8546d42bae8"
server_key_version: "kuduclusterkey@0"
I20260619 12:31:38.824262 11988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 12:31:38.857125 12003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:31:38.859053 11988 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.002s sys 0.000s
I20260619 12:31:38.859212 11988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "6e195552ad2b405d960744b5c002e84a"
format_stamp: "Formatted at 2026-06-19 12:31:38 on dist-test-slave-fg3h"
server_key: "514c10063d857aca3e4aea5745a24f22"
server_key_iv: "2e347a7e663a1ef0b928f8546d42bae8"
server_key_version: "kuduclusterkey@0"
I20260619 12:31:38.859359 11988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:31:38.875363 11988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:31:38.888031 11988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:31:38.888306 11988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:31:38.889010 11988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:31:38.890096 11988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:31:38.890167 11988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:31:38.890221 11988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:31:38.890250 11988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:31:38.901224 11988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42351
I20260619 12:31:38.901386 12116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42351 every 8 connection(s)
I20260619 12:31:38.902339 11988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:31:38.907464 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 11988
I20260619 12:31:38.907580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:31:38.907838 392 external_mini_cluster.cc:1428] Setting key 7b663a2c17af50e01460c07d6f886508
Jun 19 12:31:38 dist-test-slave-fg3h krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:31:38.917897 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:38.904412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46919 (local address 127.0.98.62:45979)
0619 12:31:38.904602 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 12:31:38.904606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:31:38.905826 (+ 1220us) server_negotiation.cc:407] Connection header received
0619 12:31:38.907084 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:31:38.907089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:31:38.907154 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:31:38.907243 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:31:38.908700 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.909498 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.911052 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.911402 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.911509 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:31:38.914133 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:31:38.914137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:31:38.914140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:31:38.914170 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:31:38.915704 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:31:38.916266 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:31:38.916271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:31:38.916273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:31:38.916352 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:31:38.916701 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:31:38.916706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:31:38.916708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:31:38.916892 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:31:38.917001 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 12:31:38.917614 (+ 613us) server_negotiation.cc:299] Negotiation successful
0619 12:31:38.917736 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 12:31:38.918753 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:38.904924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45979 (local address 127.0.98.3:46919)
0619 12:31:38.905619 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0619 12:31:38.905717 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0619 12:31:38.906831 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:31:38.907540 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:31:38.907549 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:31:38.907877 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:31:38.908535 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:31:38.908555 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.909651 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.909654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:31:38.910926 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:31:38.910937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.911564 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.911567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:31:38.911752 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:31:38.912450 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:31:38.912471 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:31:38.913966 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:31:38.915865 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:31:38.915871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:31:38.915889 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:31:38.916158 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:31:38.916453 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:31:38.916456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:31:38.916458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:31:38.916609 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:31:38.916999 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:31:38.917006 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:31:38.917342 (+ 336us) client_negotiation.cc:769] Sending connection context
0619 12:31:38.917562 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 12:31:38.917897 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":191,"threads_started":1}
I20260619 12:31:38.919950 12117 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45979
I20260619 12:31:38.920215 12117 heartbeater.cc:464] Registering TS with master...
I20260619 12:31:38.920889 12117 heartbeater.cc:511] Master 127.0.98.62:45979 requested a full tablet report, sending...
I20260619 12:31:38.921972 11633 ts_manager.cc:194] Registered new tserver with Master: 6e195552ad2b405d960744b5c002e84a (127.0.98.3:42351)
I20260619 12:31:38.922742 11633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46919
I20260619 12:31:38.934418 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:31:38.945411 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:38.936628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37088 (local address 127.0.98.62:45979)
0619 12:31:38.936758 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 12:31:38.936765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:31:38.936979 (+ 214us) server_negotiation.cc:407] Connection header received
0619 12:31:38.937164 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:31:38.937167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:31:38.937224 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:31:38.937328 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:31:38.938041 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.938562 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.939480 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.939781 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.939890 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:31:38.941745 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:31:38.941748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:31:38.941751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:31:38.941778 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:31:38.943596 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:31:38.944100 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:31:38.944104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:31:38.944106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:31:38.944193 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:31:38.944481 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:31:38.944484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:31:38.944486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:31:38.944712 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:31:38.944845 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 12:31:38.945116 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 12:31:38.945240 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 12:31:38.948660 11633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:31:38.952796 11633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:31:38.970484 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:38.962483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45207 (local address 127.0.0.1:57544)
0619 12:31:38.964383 (+ 1900us) negotiation.cc:107] Waiting for socket to connect
0619 12:31:38.964459 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 12:31:38.964608 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:31:38.965443 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:31:38.965449 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:31:38.965473 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:31:38.965604 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:31:38.965611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.967056 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.967060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:31:38.968520 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:31:38.968531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.969894 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.969898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:31:38.970016 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:31:38.970056 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 12:31:38.970185 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 12:31:38.970318 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1719,"thread_start_us":113,"threads_started":1}
I20260619 12:31:38.971539 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:38.964177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52586 (local address 127.0.98.3:42351)
0619 12:31:38.964695 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0619 12:31:38.964707 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:31:38.964731 (+ 24us) server_negotiation.cc:407] Connection header received
0619 12:31:38.964875 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:31:38.964880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:31:38.965874 (+ 994us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:31:38.966020 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:31:38.966388 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.967579 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.969783 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.970544 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.970640 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:31:38.970768 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 12:31:38.971310 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 12:31:38.971415 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":156,"threads_started":1}
I20260619 12:31:38.971210 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:38.964018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42351 (local address 127.0.0.1:52586)
0619 12:31:38.964570 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0619 12:31:38.964581 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:31:38.964658 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:31:38.966154 (+ 1496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:31:38.966157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:31:38.966174 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:31:38.966256 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:31:38.966262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.967707 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.967711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:31:38.969632 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:31:38.969642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.970837 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.970841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:31:38.970923 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:31:38.970943 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 12:31:38.971009 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 12:31:38.971075 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":91,"threads_started":1}
I20260619 12:31:38.972407 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:38.962612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57544 (local address 127.0.98.1:45207)
0619 12:31:38.964106 (+ 1494us) server_negotiation.cc:206] Beginning negotiation
0619 12:31:38.964114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:31:38.964750 (+ 636us) server_negotiation.cc:407] Connection header received
0619 12:31:38.964851 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:31:38.964857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:31:38.965108 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:31:38.965270 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:31:38.965737 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.966924 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.968667 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.969569 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.969676 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:31:38.969827 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 12:31:38.972128 (+ 2301us) server_negotiation.cc:299] Negotiation successful
0619 12:31:38.972263 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1376,"thread_start_us":115,"threads_started":1}
I20260619 12:31:38.975097 11774 tablet_service.cc:1467] Processing CreateTablet for tablet 4cb53bf67f0d47bda180b77b3e440640 (DEFAULT_TABLE table=test-table [id=f7aa7cd39c4d40a09899bebc6891e3d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:31:38.975045 12051 tablet_service.cc:1467] Processing CreateTablet for tablet 4cb53bf67f0d47bda180b77b3e440640 (DEFAULT_TABLE table=test-table [id=f7aa7cd39c4d40a09899bebc6891e3d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:31:38.976227 12051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cb53bf67f0d47bda180b77b3e440640. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:31:38.976227 11774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cb53bf67f0d47bda180b77b3e440640. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:31:38.984962 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:38.964328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45735 (local address 127.0.0.1:53992)
0619 12:31:38.971913 (+ 7585us) negotiation.cc:107] Waiting for socket to connect
0619 12:31:38.971928 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:31:38.972007 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:31:38.975465 (+ 3458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:31:38.975470 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:31:38.975489 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:31:38.975698 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:31:38.975709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.979359 (+ 3650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.979362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:31:38.980563 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:31:38.980574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.984502 (+ 3928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.984506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:31:38.984612 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:31:38.984645 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 12:31:38.984711 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 12:31:38.984802 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7217,"thread_start_us":98,"threads_started":1}
I20260619 12:31:38.985487 12136 tablet_bootstrap.cc:492] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a: Bootstrap starting.
I20260619 12:31:38.985899 12135 tablet_bootstrap.cc:492] T 4cb53bf67f0d47bda180b77b3e440640 P 9f84da732c7c40559d26bcbbdb099854: Bootstrap starting.
I20260619 12:31:38.985997 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:38.964692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.0.98.2:45735)
0619 12:31:38.974759 (+ 10067us) server_negotiation.cc:206] Beginning negotiation
0619 12:31:38.974773 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:31:38.974802 (+ 29us) server_negotiation.cc:407] Connection header received
0619 12:31:38.974884 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:31:38.974889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:31:38.975108 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:31:38.975293 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:31:38.975846 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.977197 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.980745 (+ 3548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:38.981606 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:38.985439 (+ 3833us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:31:38.985685 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 12:31:38.985744 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 12:31:38.985852 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9953,"thread_start_us":124,"threads_started":1}
I20260619 12:31:38.988643 11901 tablet_service.cc:1467] Processing CreateTablet for tablet 4cb53bf67f0d47bda180b77b3e440640 (DEFAULT_TABLE table=test-table [id=f7aa7cd39c4d40a09899bebc6891e3d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:31:38.989725 11901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cb53bf67f0d47bda180b77b3e440640. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:31:38.994268 12135 tablet_bootstrap.cc:654] T 4cb53bf67f0d47bda180b77b3e440640 P 9f84da732c7c40559d26bcbbdb099854: Neither blocks nor log segments found. Creating new log.
I20260619 12:31:38.995195 12135 log.cc:824] T 4cb53bf67f0d47bda180b77b3e440640 P 9f84da732c7c40559d26bcbbdb099854: Log is configured to *not* fsync() on all Append() calls
I20260619 12:31:38.996387 12136 tablet_bootstrap.cc:654] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a: Neither blocks nor log segments found. Creating new log.
I20260619 12:31:38.997313 12136 log.cc:824] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a: Log is configured to *not* fsync() on all Append() calls
I20260619 12:31:38.999234 12135 tablet_bootstrap.cc:492] T 4cb53bf67f0d47bda180b77b3e440640 P 9f84da732c7c40559d26bcbbdb099854: No bootstrap required, opened a new log
I20260619 12:31:38.999457 12135 ts_tablet_manager.cc:1397] T 4cb53bf67f0d47bda180b77b3e440640 P 9f84da732c7c40559d26bcbbdb099854: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20260619 12:31:39.002907 12135 raft_consensus.cc:348] T 4cb53bf67f0d47bda180b77b3e440640 P 9f84da732c7c40559d26bcbbdb099854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f84da732c7c40559d26bcbbdb099854" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45207 } } peers { permanent_uuid: "6e195552ad2b405d960744b5c002e84a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } } peers { permanent_uuid: "ca69e4e481d445219451fb703f7cd92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45735 } }
I20260619 12:31:39.003171 12135 raft_consensus.cc:374] T 4cb53bf67f0d47bda180b77b3e440640 P 9f84da732c7c40559d26bcbbdb099854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:31:39.003230 12135 raft_consensus.cc:729] T 4cb53bf67f0d47bda180b77b3e440640 P 9f84da732c7c40559d26bcbbdb099854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f84da732c7c40559d26bcbbdb099854, State: Initialized, Role: FOLLOWER
I20260619 12:31:39.003741 12135 consensus_queue.cc:260] T 4cb53bf67f0d47bda180b77b3e440640 P 9f84da732c7c40559d26bcbbdb099854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f84da732c7c40559d26bcbbdb099854" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45207 } } peers { permanent_uuid: "6e195552ad2b405d960744b5c002e84a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } } peers { permanent_uuid: "ca69e4e481d445219451fb703f7cd92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45735 } }
I20260619 12:31:39.004901 12135 ts_tablet_manager.cc:1428] T 4cb53bf67f0d47bda180b77b3e440640 P 9f84da732c7c40559d26bcbbdb099854: Time spent starting tablet: real 0.005s user 0.005s sys 0.002s
I20260619 12:31:39.005278 11847 heartbeater.cc:503] Master 127.0.98.62:45979 was elected leader, sending a full tablet report...
I20260619 12:31:39.005652 12138 tablet_bootstrap.cc:492] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b: Bootstrap starting.
I20260619 12:31:39.006381 12136 tablet_bootstrap.cc:492] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a: No bootstrap required, opened a new log
I20260619 12:31:39.006589 12136 ts_tablet_manager.cc:1397] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.004s
I20260619 12:31:39.010850 12138 tablet_bootstrap.cc:654] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b: Neither blocks nor log segments found. Creating new log.
I20260619 12:31:39.011626 12136 raft_consensus.cc:348] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f84da732c7c40559d26bcbbdb099854" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45207 } } peers { permanent_uuid: "6e195552ad2b405d960744b5c002e84a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } } peers { permanent_uuid: "ca69e4e481d445219451fb703f7cd92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45735 } }
I20260619 12:31:39.011911 12136 raft_consensus.cc:374] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:31:39.011973 12136 raft_consensus.cc:729] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e195552ad2b405d960744b5c002e84a, State: Initialized, Role: FOLLOWER
I20260619 12:31:39.012418 12138 log.cc:824] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b: Log is configured to *not* fsync() on all Append() calls
I20260619 12:31:39.012493 12136 consensus_queue.cc:260] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f84da732c7c40559d26bcbbdb099854" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45207 } } peers { permanent_uuid: "6e195552ad2b405d960744b5c002e84a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } } peers { permanent_uuid: "ca69e4e481d445219451fb703f7cd92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45735 } }
I20260619 12:31:39.013598 12136 ts_tablet_manager.cc:1428] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 12:31:39.013705 12117 heartbeater.cc:503] Master 127.0.98.62:45979 was elected leader, sending a full tablet report...
I20260619 12:31:39.017058 12138 tablet_bootstrap.cc:492] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b: No bootstrap required, opened a new log
I20260619 12:31:39.017330 12138 ts_tablet_manager.cc:1397] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260619 12:31:39.020174 12138 raft_consensus.cc:348] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f84da732c7c40559d26bcbbdb099854" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45207 } } peers { permanent_uuid: "6e195552ad2b405d960744b5c002e84a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } } peers { permanent_uuid: "ca69e4e481d445219451fb703f7cd92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45735 } }
I20260619 12:31:39.020448 12138 raft_consensus.cc:374] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:31:39.020514 12138 raft_consensus.cc:729] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca69e4e481d445219451fb703f7cd92b, State: Initialized, Role: FOLLOWER
I20260619 12:31:39.020987 12138 consensus_queue.cc:260] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f84da732c7c40559d26bcbbdb099854" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45207 } } peers { permanent_uuid: "6e195552ad2b405d960744b5c002e84a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } } peers { permanent_uuid: "ca69e4e481d445219451fb703f7cd92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45735 } }
I20260619 12:31:39.021660 11983 heartbeater.cc:503] Master 127.0.98.62:45979 was elected leader, sending a full tablet report...
I20260619 12:31:39.022032 12138 ts_tablet_manager.cc:1428] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260619 12:31:39.044271 12141 raft_consensus.cc:482] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:31:39.044510 12141 raft_consensus.cc:504] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f84da732c7c40559d26bcbbdb099854" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45207 } } peers { permanent_uuid: "6e195552ad2b405d960744b5c002e84a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } } peers { permanent_uuid: "ca69e4e481d445219451fb703f7cd92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45735 } }
I20260619 12:31:39.045756 12141 leader_election.cc:288] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f84da732c7c40559d26bcbbdb099854 (127.0.98.1:45207), ca69e4e481d445219451fb703f7cd92b (127.0.98.2:45735)
W20260619 12:31:39.047010 11984 tablet.cc:2369] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:31:39.050750 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:39.045948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45207 (local address 127.0.98.3:39149)
0619 12:31:39.046305 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 12:31:39.046323 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:31:39.046415 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:31:39.046748 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:31:39.046751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:31:39.046769 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:31:39.046887 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:31:39.046895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:31:39.047863 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:31:39.047866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:31:39.049155 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:31:39.049165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:31:39.050338 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:31:39.050341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:31:39.050455 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:31:39.050476 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 12:31:39.050544 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 12:31:39.050611 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258}
I20260619 12:31:39.051271 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:39.046140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39149 (local address 127.0.98.1:45207)
0619 12:31:39.046543 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 12:31:39.046547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:31:39.046558 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:31:39.046601 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:31:39.046603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:31:39.046642 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:31:39.046735 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:31:39.047019 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:39.047729 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:39.049444 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:39.050091 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:39.050195 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:31:39.050280 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 12:31:39.051036 (+ 756us) server_negotiation.cc:299] Negotiation successful
0619 12:31:39.051115 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315}
I20260619 12:31:39.052035 11786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb53bf67f0d47bda180b77b3e440640" candidate_uuid: "6e195552ad2b405d960744b5c002e84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f84da732c7c40559d26bcbbdb099854" is_pre_election: true
I20260619 12:31:39.052413 11786 raft_consensus.cc:2454] T 4cb53bf67f0d47bda180b77b3e440640 P 9f84da732c7c40559d26bcbbdb099854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e195552ad2b405d960744b5c002e84a in term 0.
I20260619 12:31:39.052490 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:39.046372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58605 (local address 127.0.98.2:45735)
0619 12:31:39.046518 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 12:31:39.046524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:31:39.048123 (+ 1599us) server_negotiation.cc:407] Connection header received
0619 12:31:39.048305 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:31:39.048309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:31:39.048354 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:31:39.048400 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:31:39.048753 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:39.049542 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:39.051208 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:39.051810 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:39.051888 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:31:39.051946 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 12:31:39.052286 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 12:31:39.052362 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 12:31:39.052932 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:39.046192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45735 (local address 127.0.98.3:58605)
0619 12:31:39.048043 (+ 1851us) negotiation.cc:107] Waiting for socket to connect
0619 12:31:39.048056 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:31:39.048183 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:31:39.048518 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:31:39.048520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:31:39.048535 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:31:39.048634 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:31:39.048639 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:31:39.049692 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:31:39.049695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:31:39.051080 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:31:39.051089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:31:39.052041 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:31:39.052045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:31:39.052139 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:31:39.052166 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 12:31:39.052660 (+ 494us) client_negotiation.cc:240] Negotiation successful
0619 12:31:39.052769 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1793,"mutex_wait_us":1444,"thread_start_us":130,"threads_started":1}
I20260619 12:31:39.053341 12004 leader_election.cc:302] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e195552ad2b405d960744b5c002e84a, 9f84da732c7c40559d26bcbbdb099854; no voters:
I20260619 12:31:39.053520 11924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb53bf67f0d47bda180b77b3e440640" candidate_uuid: "6e195552ad2b405d960744b5c002e84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca69e4e481d445219451fb703f7cd92b" is_pre_election: true
I20260619 12:31:39.053647 12141 raft_consensus.cc:2781] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:31:39.053721 12141 raft_consensus.cc:482] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:31:39.053758 12141 raft_consensus.cc:3037] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:31:39.053797 11924 raft_consensus.cc:2454] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e195552ad2b405d960744b5c002e84a in term 0.
I20260619 12:31:39.054931 12141 raft_consensus.cc:504] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f84da732c7c40559d26bcbbdb099854" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45207 } } peers { permanent_uuid: "6e195552ad2b405d960744b5c002e84a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } } peers { permanent_uuid: "ca69e4e481d445219451fb703f7cd92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45735 } }
I20260619 12:31:39.055269 12141 leader_election.cc:288] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [CANDIDATE]: Term 1 election: Requested vote from peers 9f84da732c7c40559d26bcbbdb099854 (127.0.98.1:45207), ca69e4e481d445219451fb703f7cd92b (127.0.98.2:45735)
I20260619 12:31:39.055668 11924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb53bf67f0d47bda180b77b3e440640" candidate_uuid: "6e195552ad2b405d960744b5c002e84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca69e4e481d445219451fb703f7cd92b"
I20260619 12:31:39.055713 11786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb53bf67f0d47bda180b77b3e440640" candidate_uuid: "6e195552ad2b405d960744b5c002e84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f84da732c7c40559d26bcbbdb099854"
I20260619 12:31:39.055792 11924 raft_consensus.cc:3037] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:31:39.055835 11786 raft_consensus.cc:3037] T 4cb53bf67f0d47bda180b77b3e440640 P 9f84da732c7c40559d26bcbbdb099854 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:31:39.057078 11786 raft_consensus.cc:2454] T 4cb53bf67f0d47bda180b77b3e440640 P 9f84da732c7c40559d26bcbbdb099854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e195552ad2b405d960744b5c002e84a in term 1.
I20260619 12:31:39.057078 11924 raft_consensus.cc:2454] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e195552ad2b405d960744b5c002e84a in term 1.
I20260619 12:31:39.057446 12007 leader_election.cc:302] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e195552ad2b405d960744b5c002e84a, ca69e4e481d445219451fb703f7cd92b; no voters:
I20260619 12:31:39.057794 12141 raft_consensus.cc:2781] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:31:39.058046 12141 raft_consensus.cc:686] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [term 1 LEADER]: Becoming Leader. State: Replica: 6e195552ad2b405d960744b5c002e84a, State: Running, Role: LEADER
I20260619 12:31:39.058399 12141 consensus_queue.cc:237] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f84da732c7c40559d26bcbbdb099854" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45207 } } peers { permanent_uuid: "6e195552ad2b405d960744b5c002e84a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } } peers { permanent_uuid: "ca69e4e481d445219451fb703f7cd92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45735 } }
I20260619 12:31:39.061563 11632 catalog_manager.cc:5495] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e195552ad2b405d960744b5c002e84a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6e195552ad2b405d960744b5c002e84a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f84da732c7c40559d26bcbbdb099854" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e195552ad2b405d960744b5c002e84a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca69e4e481d445219451fb703f7cd92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45735 } health_report { overall_health: UNKNOWN } } }
I20260619 12:31:39.095619 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:39.090150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52594 (local address 127.0.98.3:42351)
0619 12:31:39.090701 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0619 12:31:39.090706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:31:39.090724 (+ 18us) server_negotiation.cc:407] Connection header received
0619 12:31:39.090818 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:31:39.090821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:31:39.090893 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:31:39.091022 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:31:39.091427 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:39.092185 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:39.093447 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:31:39.093799 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:31:39.093891 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:31:39.094319 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:31:39.094899 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:31:39.095012 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 12:31:39.095422 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 12:31:39.095481 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 12:31:39.102643 11924 raft_consensus.cc:1261] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b [term 1 FOLLOWER]: Refusing update from remote peer 6e195552ad2b405d960744b5c002e84a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:31:39.102748 11786 raft_consensus.cc:1261] T 4cb53bf67f0d47bda180b77b3e440640 P 9f84da732c7c40559d26bcbbdb099854 [term 1 FOLLOWER]: Refusing update from remote peer 6e195552ad2b405d960744b5c002e84a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:31:39.103483 12141 consensus_queue.cc:1035] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f84da732c7c40559d26bcbbdb099854" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:31:39.103735 12145 consensus_queue.cc:1035] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca69e4e481d445219451fb703f7cd92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Jun 19 12:31:39 dist-test-slave-fg3h krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:31:39.124738 12149 mvcc.cc:205] Tried to move back new op lower bound from 7298548937116327936 to 7298548936946466816. Current Snapshot: MvccSnapshot[applied={T|T < 7298548937116327936}]
I20260619 12:31:39.125895 12150 mvcc.cc:205] Tried to move back new op lower bound from 7298548937116327936 to 7298548936946466816. Current Snapshot: MvccSnapshot[applied={T|T < 7298548937116327936}]
I20260619 12:31:39.126082 12151 mvcc.cc:205] Tried to move back new op lower bound from 7298548937116327936 to 7298548936946466816. Current Snapshot: MvccSnapshot[applied={T|T < 7298548937116327936}]
I20260619 12:31:39.138209 11632 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:31:39.138406 11632 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:31:39.140993 11632 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 190d743b4f1147da901cfda7725cea1e: Sending DeleteTablet for 3 replicas of tablet 4cb53bf67f0d47bda180b77b3e440640
I20260619 12:31:39.142027 12051 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cb53bf67f0d47bda180b77b3e440640 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:31:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52586
I20260619 12:31:39.145267 11774 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cb53bf67f0d47bda180b77b3e440640 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:31:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57544
I20260619 12:31:39.146075 11901 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cb53bf67f0d47bda180b77b3e440640 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:31:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53992
I20260619 12:31:39.146878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 11714
I20260619 12:31:39.150352 12161 tablet_replica.cc:330] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b: stopping tablet replica
I20260619 12:31:39.150764 12161 raft_consensus.cc:2229] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:31:39.156495 12161 raft_consensus.cc:2258] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:31:39.156864 12159 tablet_replica.cc:330] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a: stopping tablet replica
I20260619 12:31:39.157507 12159 raft_consensus.cc:2229] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:31:39.159161 12161 ts_tablet_manager.cc:1905] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:31:39.159695 12159 raft_consensus.cc:2258] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:31:39.163408 12161 ts_tablet_manager.cc:1918] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:31:39.164757 12159 ts_tablet_manager.cc:1905] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:31:39.167776 12159 ts_tablet_manager.cc:1918] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260619 12:31:39.168190 11617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45207 (error 108)
I20260619 12:31:39.169302 12161 log.cc:1196] T 4cb53bf67f0d47bda180b77b3e440640 P ca69e4e481d445219451fb703f7cd92b: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/4cb53bf67f0d47bda180b77b3e440640
I20260619 12:31:39.169577 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 11853
I20260619 12:31:39.177196 12159 log.cc:1196] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/4cb53bf67f0d47bda180b77b3e440640
I20260619 12:31:39.177724 12159 ts_tablet_manager.cc:1939] T 4cb53bf67f0d47bda180b77b3e440640 P 6e195552ad2b405d960744b5c002e84a: Deleting consensus metadata
I20260619 12:31:39.179229 11620 catalog_manager.cc:4841] TS 6e195552ad2b405d960744b5c002e84a (127.0.98.3:42351): tablet 4cb53bf67f0d47bda180b77b3e440640 (table test-table [id=f7aa7cd39c4d40a09899bebc6891e3d5]) successfully deleted
I20260619 12:31:39.180354 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:31:39.180037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45207 (local address 127.0.0.1:57560)
0619 12:31:39.180180 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 12:31:39.180263 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 12:31:39.180508 11617 catalog_manager.cc:4584] TS 9f84da732c7c40559d26bcbbdb099854 (127.0.98.1:45207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4cb53bf67f0d47bda180b77b3e440640: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45207: connect: Connection refused (error 111)
W20260619 12:31:39.183312 12129 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:31:39.182606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45735 (local address 127.0.0.1:54004)
0619 12:31:39.182764 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 12:31:39.182779 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:31:39.182884 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:31:39.183213 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45735: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 12:31:39.183982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 11988
I20260619 12:31:39.192226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 11601
I20260619 12:31:39.203604 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 12:31:39.548736 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 12:31:39.548952 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 12:31:39.894333 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 12:31:39.894 UTC [11187] LOG: received smart shutdown request
2026-06-19 12:31:39.898 UTC [11187] LOG: background worker "logical replication launcher" (PID 11196) exited with exit code 1
2026-06-19 12:31:39.898 UTC [11191] LOG: shutting down
2026-06-19 12:31:40.070 UTC [11187] LOG: database system is shut down
2026-06-19T12:31:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46568 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:31:40 dist-test-slave-fg3h krb5kdc[12172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:31:40 dist-test-slave-fg3h krb5kdc[12172](info): set up 2 sockets
Jun 19 12:31:40 dist-test-slave-fg3h krb5kdc[12172](info): commencing operation
krb5kdc: starting...
W20260619 12:31:41.281217 392 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:31:41 dist-test-slave-fg3h krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:31:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:31:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 12:31:41.447351 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskf39FOk/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/postgres -l logfile start
2026-06-19 12:31:43.972 UTC [12202] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 12:31:43.973 UTC [12202] LOG: listening on IPv4 address "127.0.98.20", port 36817
2026-06-19 12:31:43.977 UTC [12202] LOG: listening on Unix socket "/tmp/.s.PGSQL.36817"
2026-06-19 12:31:43.997 UTC [12205] LOG: database system was shut down at 2026-06-19 12:31:42 UTC
2026-06-19 12:31:44.001 UTC [12202] LOG: database system is ready to accept connections
I20260619 12:31:44.960878 392 mini_postgres.cc:96] Postgres bound to 36817
2026-06-19 12:31:44.967 UTC [12213] FATAL: database "slave" does not exist
127.0.98.20:36817 - accepting connections
I20260619 12:31:44.967907 392 mini_ranger.cc:161] Starting Ranger...
I20260619 12:31:44.982982 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 12:31:45.471300 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 12:31:45.471370 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ranger-admin
2026-06-19 12:31:45,747 [I] DB FLAVOR :POSTGRES
2026-06-19 12:31:45,748 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 12:31:45,748 [I] Checking connection..
2026-06-19 12:31:45,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 12:31:45,986 [I] Checking connection passed.
2026-06-19 12:31:45,986 [I] --------- Verifying version history table ---------
2026-06-19 12:31:45,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 12:31:46,237 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 12:31:46,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 12:31:46,487 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 12:31:46,488 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 12:31:46,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 12:31:46.726 UTC [12277] WARNING: there is no transaction in progress
2026-06-19 12:31:46,732 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 12:31:46,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 12:31:46,996 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 12:31:46,996 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 12:31:46,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 12:31:47,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 12:31:47,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fg3h.c.gcp-upstream.internal','N') ;"
2026-06-19 12:31:47,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 12:31:47,952 [I] Table x_portal_user does not exist in database ranger
2026-06-19 12:31:47,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 12:31:48,185 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 12:31:48,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 12:31:48,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 12:31:48,643 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 12:31:48,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 12:31:49.640 UTC [12385] WARNING: there is no transaction in progress
2026-06-19 12:31:49.657 UTC [12385] WARNING: there is no transaction in progress
2026-06-19 12:31:49.673 UTC [12385] WARNING: there is no transaction in progress
2026-06-19 12:31:49.689 UTC [12385] WARNING: there is no transaction in progress
2026-06-19 12:31:49.705 UTC [12385] WARNING: there is no transaction in progress
2026-06-19 12:31:49.726 UTC [12385] WARNING: there is no transaction in progress
2026-06-19 12:31:49.780 UTC [12385] WARNING: there is no transaction in progress
2026-06-19 12:31:49.787 UTC [12385] WARNING: there is no transaction in progress
2026-06-19 12:31:49.801 UTC [12385] WARNING: there is no transaction in progress
2026-06-19 12:31:49.812 UTC [12385] WARNING: there is no transaction in progress
2026-06-19 12:31:49.822 UTC [12385] WARNING: there is no transaction in progress
2026-06-19 12:31:49.836 UTC [12385] WARNING: there is no transaction in progress
2026-06-19 12:31:49.846 UTC [12385] WARNING: there is no transaction in progress
2026-06-19 12:31:49.854 UTC [12385] WARNING: there is no transaction in progress
2026-06-19 12:31:49.862 UTC [12385] WARNING: there is no transaction in progress
2026-06-19 12:31:49.869 UTC [12385] WARNING: there is no transaction in progress
2026-06-19 12:31:50,537 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 12:31:50,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 12:31:50,767 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 12:31:50,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 12:31:51,010 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 12:31:51,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 12:31:51,238 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 12:31:51,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 12:31:51,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 12:31:51,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fg3h.c.gcp-upstream.internal';"
2026-06-19 12:31:51,973 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 12:31:51,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 12:31:52,213 [I] DB_PATCHES have already been applied
I20260619 12:31:52.219314 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskf39FOk/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 12:31:52.219377 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 12:31:52.222766 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:51087
Jun 19, 2026 12:31:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 12:31:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 12:31:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 12:31:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 12:31:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 12:31:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 12:31:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 12:31:53.340248 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.372s user 0.003s sys 0.007s
Jun 19, 2026 12:32:14 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 12:32:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 12:32:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 12:32:14 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 12:32:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 12:32:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 12:32:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 12:32:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 12:32:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 12:32:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 12:32:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 12:32:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 12:32:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 12:32:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 12:32:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 12:32:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 12:32:14 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 12:32:15 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 12:32:15.641448 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 12:32:15 dist-test-slave-fg3h krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872335, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 12:32:15.744947 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 12:32:15.757359 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 12:32:16.718467 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 12:32:16.718559 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ranger-kms
I20260619 12:32:16.718581 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:36817
2026-06-19 12:32:16,891 [I] DB FLAVOR :POSTGRES
2026-06-19 12:32:16,891 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 12:32:16,891 [I] Checking connection
2026-06-19 12:32:16,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 12:32:17,147 [I] connection success
2026-06-19 12:32:17,148 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 12:32:17,148 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 12:32:17,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 12:32:17,443 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 12:32:17,444 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 12:32:17,444 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 12:32:17,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36817/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 12:32:17,727 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 12:32:18.206570 392 mini_ranger_kms.cc:310] Created kms service
I20260619 12:32:18.384146 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 12:32:18.427652 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 12:32:18 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872335, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:32:18.809187 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 12:32:18.809350 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskf39FOk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskf39FOk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 12:32:18.809374 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 12:32:18.813388 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12565
I20260619 12:32:18.813480 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:45199
Jun 19, 2026 12:32:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 12:32:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 12:32:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 12:32:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 12:32:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 12:32:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 12:32:21.092862 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.348s user 0.010s sys 0.145s
I20260619 12:32:21.092988 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 12:32:21.093053 392 mini_ranger_kms.cc:401] 127.0.98.20:45199/kms/v1/keys
Jun 19 12:32:21 dist-test-slave-fg3h krb5kdc[12172](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781872341, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 12:32:21 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872341, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 12:32:22 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872341, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 12:32:24 dist-test-slave-fg3h krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:32:24.271646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39711
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39711
--ranger_config_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45199/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:32:24.404801 12616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:24.405076 12616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:24.405194 12616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:24.409451 12616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:32:24.409538 12616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:24.409559 12616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:24.409605 12616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:32:24.409682 12616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:32:24.415405 12616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:45199/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39711
--ranger_config_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:24.416494 12616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:24.417624 12616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:24.426558 12621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:24.426609 12616 server_base.cc:1034] running on GCE node
W20260619 12:32:24.426793 12622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:24.426537 12624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:24.427414 12616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:24.428622 12616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:24.429832 12616 hybrid_clock.cc:648] HybridClock initialized: now 1781872344429814 us; error 38 us; skew 500 ppm
Jun 19 12:32:24 dist-test-slave-fg3h krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:24.437513 12616 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:32:24.438639 12616 webserver.cc:460] Webserver started at http://127.0.98.62:45337/ using document root <none> and password file <none>
I20260619 12:32:24.439244 12616 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:24.439302 12616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:24.439524 12616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:32:24 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:32:24.500519 12616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "1aa47aac4dde42f49fb90523119d52f3"
format_stamp: "Formatted at 2026-06-19 12:32:24 on dist-test-slave-fg3h"
server_key: "16e457096d765166564e1a1df66b5b04"
server_key_iv: "28d6865d12f8c3b261f2f0fbf400851f"
server_key_version: "kuduclusterkey@0"
I20260619 12:32:24.501328 12616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "1aa47aac4dde42f49fb90523119d52f3"
format_stamp: "Formatted at 2026-06-19 12:32:24 on dist-test-slave-fg3h"
server_key: "16e457096d765166564e1a1df66b5b04"
server_key_iv: "28d6865d12f8c3b261f2f0fbf400851f"
server_key_version: "kuduclusterkey@0"
I20260619 12:32:24.506332 12616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 12:32:24.524938 12631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:24.526826 12616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:32:24.526983 12616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "1aa47aac4dde42f49fb90523119d52f3"
format_stamp: "Formatted at 2026-06-19 12:32:24 on dist-test-slave-fg3h"
server_key: "16e457096d765166564e1a1df66b5b04"
server_key_iv: "28d6865d12f8c3b261f2f0fbf400851f"
server_key_version: "kuduclusterkey@0"
I20260619 12:32:24.527132 12616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:24.562593 12616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:24.565901 12616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:24.566090 12616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:24.574509 12616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39711
I20260619 12:32:24.574520 12683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39711 every 8 connection(s)
I20260619 12:32:24.575742 12616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:32:24.578702 12684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:32:24.580945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 12616
I20260619 12:32:24.581074 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:32:24.581434 392 external_mini_cluster.cc:1428] Setting key 3cce7d23475c7b4c7c643037dc41712e
I20260619 12:32:24.585062 12684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3: Bootstrap starting.
I20260619 12:32:24.587536 12684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3: Neither blocks nor log segments found. Creating new log.
I20260619 12:32:24.588485 12684 log.cc:824] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:24.590788 12684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3: No bootstrap required, opened a new log
Jun 19 12:32:24 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:32:24.595356 12684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa47aac4dde42f49fb90523119d52f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39711 } }
I20260619 12:32:24.595667 12684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:32:24.595789 12684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aa47aac4dde42f49fb90523119d52f3, State: Initialized, Role: FOLLOWER
I20260619 12:32:24.596318 12684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa47aac4dde42f49fb90523119d52f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39711 } }
I20260619 12:32:24.596477 12684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:32:24.596572 12684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:32:24.596756 12684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:32:24.597955 12684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa47aac4dde42f49fb90523119d52f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39711 } }
I20260619 12:32:24.598306 12687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:24.582894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54502 (local address 127.0.98.62:39711)
0619 12:32:24.583349 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:24.583368 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:24.583411 (+ 43us) server_negotiation.cc:407] Connection header received
0619 12:32:24.584323 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:24.584348 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:24.584782 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:24.585165 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:24.586090 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:24.587581 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:24.588848 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:24.589591 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:24.589747 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:24.592857 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:24.592864 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:24.592883 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:24.592914 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:24.594944 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:24.595475 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:24.595481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:24.595486 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:24.595614 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:24.595993 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:24.595996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:24.595998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:24.596516 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:24.596720 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:24.596998 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 12:32:24.597312 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":136,"threads_started":1}
I20260619 12:32:24.598313 12684 leader_election.cc:302] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1aa47aac4dde42f49fb90523119d52f3; no voters:
I20260619 12:32:24.598677 12684 leader_election.cc:288] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:32:24.599045 12690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:32:24.599293 12690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [term 1 LEADER]: Becoming Leader. State: Replica: 1aa47aac4dde42f49fb90523119d52f3, State: Running, Role: LEADER
I20260619 12:32:24.599642 12690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa47aac4dde42f49fb90523119d52f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39711 } }
I20260619 12:32:24.599915 12684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:32:24.601040 12690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1aa47aac4dde42f49fb90523119d52f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa47aac4dde42f49fb90523119d52f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39711 } } }
I20260619 12:32:24.601256 12690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [sys.catalog]: This master's current role is: LEADER
I20260619 12:32:24.601488 12690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1aa47aac4dde42f49fb90523119d52f3. Latest consensus state: current_term: 1 leader_uuid: "1aa47aac4dde42f49fb90523119d52f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa47aac4dde42f49fb90523119d52f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39711 } } }
I20260619 12:32:24.601625 12690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [sys.catalog]: This master's current role is: LEADER
I20260619 12:32:24.601684 12698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:32:24.604880 12698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:32:24.605399 12684 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 12:32:24.616992 12698 catalog_manager.cc:1347] Generated new cluster ID: fd4a6e22dc404afca0d6dd649a79cf3f
I20260619 12:32:24.617127 12698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:32:24.645998 12698 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:32:24.647521 12698 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:32:24.669121 12698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3: Generated new TSK 0
I20260619 12:32:24.670384 12698 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:32:25.404292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45199/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39711
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:32:25.530793 12729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:25.531178 12729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:25.531247 12729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:25.536314 12729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:25.536376 12729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:25.536473 12729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:32:25.542150 12729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:45199/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:25.543286 12729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:25.544354 12729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:25.555609 12734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:25.556042 12737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:25.556330 12735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:25.556869 12729 server_base.cc:1034] running on GCE node
I20260619 12:32:25.557401 12729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:25.558324 12729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:25.559623 12729 hybrid_clock.cc:648] HybridClock initialized: now 1781872345559603 us; error 36 us; skew 500 ppm
Jun 19 12:32:25 dist-test-slave-fg3h krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:25.563465 12729 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:32:25.564688 12729 webserver.cc:460] Webserver started at http://127.0.98.1:39381/ using document root <none> and password file <none>
I20260619 12:32:25.565462 12729 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:25.565542 12729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:25.565791 12729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:32:25 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:32:25.587679 12729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "c0aaf2534f754743be432fcfe04eb6b6"
format_stamp: "Formatted at 2026-06-19 12:32:25 on dist-test-slave-fg3h"
server_key: "1fee1989055cbd68444b9dd7d4aeb280"
server_key_iv: "40f715d376b35aabc7aa69bf4047f49e"
server_key_version: "kuduclusterkey@0"
I20260619 12:32:25.588415 12729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c0aaf2534f754743be432fcfe04eb6b6"
format_stamp: "Formatted at 2026-06-19 12:32:25 on dist-test-slave-fg3h"
server_key: "1fee1989055cbd68444b9dd7d4aeb280"
server_key_iv: "40f715d376b35aabc7aa69bf4047f49e"
server_key_version: "kuduclusterkey@0"
I20260619 12:32:25.599291 12729 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.007s sys 0.000s
I20260619 12:32:25.617806 12746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:25.619578 12729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:32:25.619817 12729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "c0aaf2534f754743be432fcfe04eb6b6"
format_stamp: "Formatted at 2026-06-19 12:32:25 on dist-test-slave-fg3h"
server_key: "1fee1989055cbd68444b9dd7d4aeb280"
server_key_iv: "40f715d376b35aabc7aa69bf4047f49e"
server_key_version: "kuduclusterkey@0"
I20260619 12:32:25.620042 12729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:25.648241 12729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:25.652175 12729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:25.652433 12729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:25.653187 12729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:32:25.654246 12729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:32:25.654356 12729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:25.654479 12729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:32:25.654552 12729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:25.666952 12729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44673
I20260619 12:32:25.669867 12729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:32:25.672374 12859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44673 every 8 connection(s)
I20260619 12:32:25.675498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 12729
I20260619 12:32:25.675634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:32:25.675989 392 external_mini_cluster.cc:1428] Setting key 35c433a32f7697426e61b7fdfe8498aa
Jun 19 12:32:25 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:32:25.692121 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:25.674274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52565 (local address 127.0.98.62:39711)
0619 12:32:25.674528 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:25.674532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:25.676543 (+ 2011us) server_negotiation.cc:407] Connection header received
0619 12:32:25.678769 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:25.678772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:25.678825 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:25.678915 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:25.681273 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:25.682195 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:25.683923 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:25.684295 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:25.684377 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:25.687630 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:25.687634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:25.687637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:25.687664 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:25.689635 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:25.690296 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:25.690299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:25.690301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:25.690389 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:25.690773 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:25.690777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:25.690778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:25.690998 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:25.691075 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:25.691774 (+ 699us) server_negotiation.cc:299] Negotiation successful
0619 12:32:25.691906 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
I20260619 12:32:25.694063 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:25.674742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39711 (local address 127.0.98.1:52565)
0619 12:32:25.676361 (+ 1619us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:25.676432 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:25.678533 (+ 2101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:25.679270 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:25.679280 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:25.679645 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:25.680399 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:25.680424 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:25.682359 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:25.682362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:25.683776 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:25.683788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:25.684592 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:25.684595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:25.684828 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:25.685601 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:25.685622 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:25.687417 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:25.689785 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:25.689792 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:25.689811 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:25.690173 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:25.690489 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:25.690492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:25.690494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:25.690668 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:25.691151 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:25.691158 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:25.691493 (+ 335us) client_negotiation.cc:769] Sending connection context
0619 12:32:25.692590 (+ 1097us) client_negotiation.cc:240] Negotiation successful
0619 12:32:25.692946 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":157,"threads_started":1}
I20260619 12:32:25.695758 12860 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39711
I20260619 12:32:25.695996 12860 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:25.696552 12860 heartbeater.cc:511] Master 127.0.98.62:39711 requested a full tablet report, sending...
I20260619 12:32:25.698356 12648 ts_manager.cc:194] Registered new tserver with Master: c0aaf2534f754743be432fcfe04eb6b6 (127.0.98.1:44673)
I20260619 12:32:25.699893 12648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:32:25.728530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45199/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39711
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:32:25 dist-test-slave-fg3h krb5kdc[12172](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781872345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 12:32:25.884038 12868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:25.884390 12868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:25.884473 12868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:25.889493 12868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:25.889611 12868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:25.889704 12868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:32:25.896114 12868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:45199/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:25.897585 12868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:25.898941 12868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:25.911468 12875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:25.911481 12874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:25.912622 12868 server_base.cc:1034] running on GCE node
W20260619 12:32:25.912987 12877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:25.913583 12868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:25.914412 12868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:25.915612 12868 hybrid_clock.cc:648] HybridClock initialized: now 1781872345915542 us; error 86 us; skew 500 ppm
Jun 19 12:32:25 dist-test-slave-fg3h krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:25.919382 12868 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:32:25.920598 12868 webserver.cc:460] Webserver started at http://127.0.98.2:38199/ using document root <none> and password file <none>
I20260619 12:32:25.921478 12868 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:25.921602 12868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:25.921903 12868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:32:25 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:32:25.938377 12868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "7060a93472474543b74b7bcc87409e16"
format_stamp: "Formatted at 2026-06-19 12:32:25 on dist-test-slave-fg3h"
server_key: "35e7d0e13c94f02c68a61078a99b4a05"
server_key_iv: "b0456a7dba14a45250fb41a490777643"
server_key_version: "kuduclusterkey@0"
I20260619 12:32:25.939232 12868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7060a93472474543b74b7bcc87409e16"
format_stamp: "Formatted at 2026-06-19 12:32:25 on dist-test-slave-fg3h"
server_key: "35e7d0e13c94f02c68a61078a99b4a05"
server_key_iv: "b0456a7dba14a45250fb41a490777643"
server_key_version: "kuduclusterkey@0"
I20260619 12:32:25.944599 12868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 12:32:25.969388 12884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:25.971268 12868 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 12:32:25.971398 12868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "7060a93472474543b74b7bcc87409e16"
format_stamp: "Formatted at 2026-06-19 12:32:25 on dist-test-slave-fg3h"
server_key: "35e7d0e13c94f02c68a61078a99b4a05"
server_key_iv: "b0456a7dba14a45250fb41a490777643"
server_key_version: "kuduclusterkey@0"
I20260619 12:32:25.971531 12868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:26.006078 12868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:26.010391 12868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:26.010672 12868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:26.011413 12868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:32:26.012562 12868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:32:26.012691 12868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:26.012796 12868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:32:26.012861 12868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:26.028215 12868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34499
I20260619 12:32:26.030323 12868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:32:26.031801 12997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34499 every 8 connection(s)
I20260619 12:32:26.037972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 12868
I20260619 12:32:26.038136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:32:26.038472 392 external_mini_cluster.cc:1428] Setting key 1fcdfacb16beda06428c3a5283b1602f
Jun 19 12:32:26 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:32:26.050616 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:26.034197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55293 (local address 127.0.98.62:39711)
0619 12:32:26.034335 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:26.034340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:26.036258 (+ 1918us) server_negotiation.cc:407] Connection header received
0619 12:32:26.037668 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:26.037672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:26.037734 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:26.037795 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:26.039534 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:26.040378 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:26.042221 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:26.042572 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:26.042656 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:26.045942 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:26.045946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:26.045949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:26.045982 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:26.047900 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:26.048601 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:26.048605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:26.048607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:26.048696 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:26.049148 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:26.049151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:26.049153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:26.049415 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:26.049500 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:26.050274 (+ 774us) server_negotiation.cc:299] Negotiation successful
0619 12:32:26.050416 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 12:32:26.052562 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:26.034768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39711 (local address 127.0.98.2:55293)
0619 12:32:26.036050 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:26.036115 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:26.037358 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:26.038251 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:26.038261 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:26.038653 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:26.039355 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:26.039375 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:26.040588 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:26.040592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:26.042056 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:26.042067 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:26.042881 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:26.042884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:26.043081 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:26.043858 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:26.043879 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:26.045704 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:26.048061 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:26.048067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:26.048089 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:26.048477 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:26.048800 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:26.048803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:26.048805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:26.049013 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:26.049594 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:26.049602 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:26.049969 (+ 367us) client_negotiation.cc:769] Sending connection context
0619 12:32:26.051058 (+ 1089us) client_negotiation.cc:240] Negotiation successful
0619 12:32:26.051453 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035,"thread_start_us":147,"threads_started":1}
I20260619 12:32:26.054456 12998 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39711
I20260619 12:32:26.054716 12998 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:26.055330 12998 heartbeater.cc:511] Master 127.0.98.62:39711 requested a full tablet report, sending...
I20260619 12:32:26.056704 12648 ts_manager.cc:194] Registered new tserver with Master: 7060a93472474543b74b7bcc87409e16 (127.0.98.2:34499)
I20260619 12:32:26.057528 12648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55293
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:32:26.088558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45199/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39711
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:32:26.278687 13003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:26.279196 13003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:26.279327 13003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:26.286362 13003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:26.286506 13003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:26.286665 13003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:32:26.295243 13003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:45199/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:26.296717 13003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:26.298164 13003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 19 12:32:26 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 12:32:26.312453 13008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:26.312665 13011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:26.313150 13003 server_base.cc:1034] running on GCE node
W20260619 12:32:26.313521 13009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:26.314085 13003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:26.314922 13003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:26.318217 13003 hybrid_clock.cc:648] HybridClock initialized: now 1781872346318121 us; error 104 us; skew 500 ppm
Jun 19 12:32:26 dist-test-slave-fg3h krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:26.321883 13003 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:32:26.322955 13003 webserver.cc:460] Webserver started at http://127.0.98.3:43073/ using document root <none> and password file <none>
I20260619 12:32:26.323628 13003 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:26.323706 13003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:26.323920 13003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 12:32:26 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:32:26.340694 13003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "19a2775f1d48400fb483448ab203a0da"
format_stamp: "Formatted at 2026-06-19 12:32:26 on dist-test-slave-fg3h"
server_key: "98b28dcd56f2c40f5d904f0b1fee88e3"
server_key_iv: "505a48ca22241b99e255ed29657e9fce"
server_key_version: "kuduclusterkey@0"
I20260619 12:32:26.341480 13003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "19a2775f1d48400fb483448ab203a0da"
format_stamp: "Formatted at 2026-06-19 12:32:26 on dist-test-slave-fg3h"
server_key: "98b28dcd56f2c40f5d904f0b1fee88e3"
server_key_iv: "505a48ca22241b99e255ed29657e9fce"
server_key_version: "kuduclusterkey@0"
I20260619 12:32:26.346837 13003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 12:32:26.364447 13018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:26.367486 13003 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260619 12:32:26.367637 13003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "19a2775f1d48400fb483448ab203a0da"
format_stamp: "Formatted at 2026-06-19 12:32:26 on dist-test-slave-fg3h"
server_key: "98b28dcd56f2c40f5d904f0b1fee88e3"
server_key_iv: "505a48ca22241b99e255ed29657e9fce"
server_key_version: "kuduclusterkey@0"
I20260619 12:32:26.367791 13003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:26.420476 13003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:26.432404 13003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:26.432682 13003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:26.433435 13003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:32:26.434512 13003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:32:26.434613 13003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:26.434723 13003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:32:26.434785 13003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:26.444681 13003 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44769
I20260619 12:32:26.445842 13003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:32:26.450484 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 13003
I20260619 12:32:26.450608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:32:26.450906 392 external_mini_cluster.cc:1428] Setting key b298a7e77cd8ee2577ba652135c4a2c9
I20260619 12:32:26.451599 13131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44769 every 8 connection(s)
Jun 19 12:32:26 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:32:26.469137 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:26.453308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46631 (local address 127.0.98.62:39711)
0619 12:32:26.453441 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:26.453446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:26.455857 (+ 2411us) server_negotiation.cc:407] Connection header received
0619 12:32:26.457069 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:26.457072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:26.457172 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:26.457224 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:26.458793 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:26.459612 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:26.461263 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:26.461591 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:26.461663 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:26.464693 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:26.464697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:26.464700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:26.464727 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:26.466531 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:26.467213 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:26.467217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:26.467219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:26.467304 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:26.467723 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:26.467727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:26.467729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:26.467958 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:26.468031 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:26.468791 (+ 760us) server_negotiation.cc:299] Negotiation successful
0619 12:32:26.468916 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 12:32:26.471050 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:26.453819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39711 (local address 127.0.98.3:46631)
0619 12:32:26.455705 (+ 1886us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:26.455763 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:26.456824 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:26.457623 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:26.457632 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:26.457977 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:26.458632 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:26.458649 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:26.459770 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:26.459773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:26.461122 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:26.461133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:26.461864 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:26.461868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:26.462054 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:26.462781 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:26.462801 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:26.464505 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:26.466675 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:26.466735 (+ 60us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:26.466753 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:26.467080 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:26.467411 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:26.467414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:26.467416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:26.467601 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:26.468115 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:26.468122 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:26.468495 (+ 373us) client_negotiation.cc:769] Sending connection context
0619 12:32:26.469622 (+ 1127us) client_negotiation.cc:240] Negotiation successful
0619 12:32:26.469975 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1657,"thread_start_us":133,"threads_started":1}
I20260619 12:32:26.472640 13132 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39711
I20260619 12:32:26.472889 13132 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:26.473526 13132 heartbeater.cc:511] Master 127.0.98.62:39711 requested a full tablet report, sending...
I20260619 12:32:26.474820 12648 ts_manager.cc:194] Registered new tserver with Master: 19a2775f1d48400fb483448ab203a0da (127.0.98.3:44769)
I20260619 12:32:26.475665 12648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46631
I20260619 12:32:26.477649 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:32:26.478529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 12729
I20260619 12:32:26.487975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 12868
I20260619 12:32:26.496135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 13003
I20260619 12:32:26.502610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 12616
I20260619 12:32:26.510651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39711
--webserver_interface=127.0.98.62
--webserver_port=45337
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39711
--ranger_config_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45199/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45199/kms with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:32:26.622889 13135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:26.623171 13135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:26.623210 13135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:26.627296 13135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:32:26.627359 13135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:26.627379 13135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:26.627395 13135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:32:26.627410 13135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:32:26.632987 13135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45199/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39711
--ranger_config_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:26.634104 13135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:26.635123 13135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:26.643702 13141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:26.643702 13140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:26.643738 13135 server_base.cc:1034] running on GCE node
W20260619 12:32:26.643715 13143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:26.644475 13135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:26.645658 13135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:26.646848 13135 hybrid_clock.cc:648] HybridClock initialized: now 1781872346646836 us; error 32 us; skew 500 ppm
Jun 19 12:32:26 dist-test-slave-fg3h krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:26.649632 13135 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:32:26.650462 13135 webserver.cc:460] Webserver started at http://127.0.98.62:45337/ using document root <none> and password file <none>
I20260619 12:32:26.650950 13135 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:26.651012 13135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 12:32:26 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:32:26.734324 13135 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:32:26.737164 13151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:26.738862 13135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:32:26.739008 13135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "1aa47aac4dde42f49fb90523119d52f3"
format_stamp: "Formatted at 2026-06-19 12:32:24 on dist-test-slave-fg3h"
server_key: "16e457096d765166564e1a1df66b5b04"
server_key_iv: "28d6865d12f8c3b261f2f0fbf400851f"
server_key_version: "kuduclusterkey@0"
I20260619 12:32:26.739763 13135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:26.769405 13135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:26.772264 13135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:26.772423 13135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:26.779898 13135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39711
I20260619 12:32:26.779924 13203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39711 every 8 connection(s)
I20260619 12:32:26.780977 13135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:32:26.785709 13204 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:32:26.787515 13204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3: Bootstrap starting.
I20260619 12:32:26.789355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 13135
I20260619 12:32:26.789863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45199/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44673
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39711
--webserver_port=39381
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45199/kms with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:32:26.805399 13204 log.cc:824] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:26.812173 13204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:32:26.812597 13204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3: Bootstrap complete.
I20260619 12:32:26.816692 13204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa47aac4dde42f49fb90523119d52f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39711 } }
I20260619 12:32:26.817165 13204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aa47aac4dde42f49fb90523119d52f3, State: Initialized, Role: FOLLOWER
I20260619 12:32:26.817710 13204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa47aac4dde42f49fb90523119d52f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39711 } }
I20260619 12:32:26.817858 13204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:32:26.817943 13204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:32:26.818064 13204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:32:26.819979 13204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa47aac4dde42f49fb90523119d52f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39711 } }
I20260619 12:32:26.820327 13204 leader_election.cc:302] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1aa47aac4dde42f49fb90523119d52f3; no voters:
I20260619 12:32:26.820686 13204 leader_election.cc:288] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 12:32:26.820775 13208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 12:32:26.821022 13208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [term 2 LEADER]: Becoming Leader. State: Replica: 1aa47aac4dde42f49fb90523119d52f3, State: Running, Role: LEADER
I20260619 12:32:26.821409 13208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa47aac4dde42f49fb90523119d52f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39711 } }
I20260619 12:32:26.821959 13204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:32:26.822345 13208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1aa47aac4dde42f49fb90523119d52f3. Latest consensus state: current_term: 2 leader_uuid: "1aa47aac4dde42f49fb90523119d52f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa47aac4dde42f49fb90523119d52f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39711 } } }
I20260619 12:32:26.822530 13208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [sys.catalog]: This master's current role is: LEADER
I20260619 12:32:26.822717 13209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1aa47aac4dde42f49fb90523119d52f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa47aac4dde42f49fb90523119d52f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39711 } } }
I20260619 12:32:26.822899 13209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3 [sys.catalog]: This master's current role is: LEADER
I20260619 12:32:26.822959 13212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:32:26.826258 13212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:32:26.826777 13204 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 12:32:26.827024 13212 catalog_manager.cc:1259] Loaded cluster ID: fd4a6e22dc404afca0d6dd649a79cf3f
I20260619 12:32:26.827087 13212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:32:26.829427 13212 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:32:26.830124 13212 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3: Loaded TSK: 0
I20260619 12:32:26.831331 13212 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 12:32:26.933203 13206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:26.933681 13206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:26.933816 13206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:26.939920 13206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:26.940079 13206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:26.940188 13206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:32:26.948318 13206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45199/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39381
--tserver_master_addrs=127.0.98.62:39711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:26.950017 13206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:26.951433 13206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:26.963791 13244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:26.964221 13242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:26.964403 13241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:26.964915 13206 server_base.cc:1034] running on GCE node
I20260619 12:32:26.965746 13206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:26.966527 13206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:26.967681 13206 hybrid_clock.cc:648] HybridClock initialized: now 1781872346967611 us; error 87 us; skew 500 ppm
Jun 19 12:32:26 dist-test-slave-fg3h krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:26.971513 13206 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:32:26.972704 13206 webserver.cc:460] Webserver started at http://127.0.98.1:39381/ using document root <none> and password file <none>
I20260619 12:32:26.973521 13206 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:26.973688 13206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 12:32:26 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:32:27.022851 13206 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.000s sys 0.003s
I20260619 12:32:27.028330 13252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:27.030205 13206 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260619 12:32:27.030376 13206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "c0aaf2534f754743be432fcfe04eb6b6"
format_stamp: "Formatted at 2026-06-19 12:32:25 on dist-test-slave-fg3h"
server_key: "1fee1989055cbd68444b9dd7d4aeb280"
server_key_iv: "40f715d376b35aabc7aa69bf4047f49e"
server_key_version: "kuduclusterkey@0"
I20260619 12:32:27.031170 13206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:27.078647 13206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:27.082154 13206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:27.082372 13206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:27.083050 13206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:32:27.084052 13206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:32:27.084117 13206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:27.084174 13206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:32:27.084233 13206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:27.096290 13206 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44673
I20260619 12:32:27.096306 13365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44673 every 8 connection(s)
I20260619 12:32:27.097711 13206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:32:27.104776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 13206
I20260619 12:32:27.105387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45199/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34499
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39711
--webserver_port=38199
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45199/kms with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:32:27 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:32:27.119104 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:27.100120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49335 (local address 127.0.98.62:39711)
0619 12:32:27.100774 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:27.100797 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:27.101541 (+ 744us) server_negotiation.cc:407] Connection header received
0619 12:32:27.102717 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:27.102727 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:27.103041 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:27.103366 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:27.104452 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:27.105683 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:27.106938 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:27.107732 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:27.107892 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:27.111468 (+ 3576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:27.111477 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:27.111500 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:27.111544 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:27.114089 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:27.114927 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:27.114937 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:27.114946 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:27.115122 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:27.115612 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:27.115618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:27.115620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:27.116223 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:27.116349 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:27.117325 (+ 976us) server_negotiation.cc:299] Negotiation successful
0619 12:32:27.117728 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":136,"threads_started":1}
I20260619 12:32:27.120909 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:27.100297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39711 (local address 127.0.98.1:49335)
0619 12:32:27.101366 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:27.101409 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:27.102269 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:27.103443 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:27.103453 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:27.103801 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:27.104270 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:27.104284 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:27.105839 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:27.105842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:27.106778 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:27.106792 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:27.108164 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:27.108169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:27.108398 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:27.109164 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:27.109191 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:27.111177 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:27.114262 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:27.114271 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:27.114292 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:27.114760 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:27.115266 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:27.115271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:27.115276 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:27.115484 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:27.116451 (+ 967us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:27.116460 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:27.116865 (+ 405us) client_negotiation.cc:769] Sending connection context
0619 12:32:27.119599 (+ 2734us) client_negotiation.cc:240] Negotiation successful
0619 12:32:27.119965 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":112,"threads_started":1}
I20260619 12:32:27.124068 13366 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39711
I20260619 12:32:27.124367 13366 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:27.125002 13366 heartbeater.cc:511] Master 127.0.98.62:39711 requested a full tablet report, sending...
W20260619 12:32:27.127449 13366 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39711 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 12:32:27.127568 13366 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:27.127755 13366 heartbeater.cc:511] Master 127.0.98.62:39711 requested a full tablet report, sending...
I20260619 12:32:27.128654 13366 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:27.128840 13366 heartbeater.cc:511] Master 127.0.98.62:39711 requested a full tablet report, sending...
W20260619 12:32:27.129663 13366 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:32:27.245580 13370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:27.245986 13370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:27.246088 13370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:27.252046 13370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:27.252203 13370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:27.252339 13370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:32:27.260717 13370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45199/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38199
--tserver_master_addrs=127.0.98.62:39711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:27.262420 13370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:27.263922 13370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:27.276525 13375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:27.277609 13376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:27.277779 13378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:27.277876 13370 server_base.cc:1034] running on GCE node
I20260619 12:32:27.278904 13370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:27.279783 13370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:27.281008 13370 hybrid_clock.cc:648] HybridClock initialized: now 1781872347280966 us; error 50 us; skew 500 ppm
Jun 19 12:32:27 dist-test-slave-fg3h krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:27.284569 13370 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:32:27.285815 13370 webserver.cc:460] Webserver started at http://127.0.98.2:38199/ using document root <none> and password file <none>
I20260619 12:32:27.286501 13370 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:27.286613 13370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 12:32:27 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:32:27.310962 13370 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 12:32:27.314376 13386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:27.316027 13370 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:32:27.316150 13370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "7060a93472474543b74b7bcc87409e16"
format_stamp: "Formatted at 2026-06-19 12:32:25 on dist-test-slave-fg3h"
server_key: "35e7d0e13c94f02c68a61078a99b4a05"
server_key_iv: "b0456a7dba14a45250fb41a490777643"
server_key_version: "kuduclusterkey@0"
I20260619 12:32:27.316843 13370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:27.345248 13370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:27.349251 13370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:27.349534 13370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:27.350261 13370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:32:27.351334 13370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:32:27.351436 13370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:27.351539 13370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:32:27.351606 13370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:27.366291 13370 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34499
I20260619 12:32:27.367695 13370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:32:27.369015 13500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34499 every 8 connection(s)
I20260619 12:32:27.374671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 13370
I20260619 12:32:27.375203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45199/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44769
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39711
--webserver_port=43073
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45199/kms with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:32:27 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:32:27.387732 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:27.371337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44855 (local address 127.0.98.62:39711)
0619 12:32:27.371498 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:27.371503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:27.373238 (+ 1735us) server_negotiation.cc:407] Connection header received
0619 12:32:27.374496 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:27.374500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:27.374557 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:27.374608 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:27.376528 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:27.377412 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:27.379171 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:27.379535 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:27.379667 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:27.382665 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:27.382669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:27.382672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:27.382705 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:27.384573 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:27.385697 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:27.385701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:27.385703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:27.385802 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:27.386263 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:27.386267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:27.386268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:27.386509 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:27.386599 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:27.387378 (+ 779us) server_negotiation.cc:299] Negotiation successful
0619 12:32:27.387532 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 12:32:27.389540 13503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:27.371846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39711 (local address 127.0.98.2:44855)
0619 12:32:27.373022 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:27.373077 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:27.374245 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:27.375081 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:27.375090 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:27.375682 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:27.376346 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:27.376364 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:27.377571 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:27.377574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:27.378985 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:27.378996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:27.379730 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:27.379734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:27.379931 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:27.380658 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:27.380679 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:27.382450 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:27.384729 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:27.384735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:27.384754 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:27.385182 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:27.385926 (+ 744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:27.385929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:27.385931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:27.386142 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:27.386673 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:27.386680 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:27.387061 (+ 381us) client_negotiation.cc:769] Sending connection context
0619 12:32:27.388004 (+ 943us) client_negotiation.cc:240] Negotiation successful
0619 12:32:27.388382 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"thread_start_us":138,"threads_started":1}
I20260619 12:32:27.391288 13501 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39711
I20260619 12:32:27.391542 13501 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:27.392171 13501 heartbeater.cc:511] Master 127.0.98.62:39711 requested a full tablet report, sending...
W20260619 12:32:27.393950 13501 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39711 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 12:32:27.394060 13501 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:27.394223 13501 heartbeater.cc:511] Master 127.0.98.62:39711 requested a full tablet report, sending...
I20260619 12:32:27.395139 13501 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:27.395292 13501 heartbeater.cc:511] Master 127.0.98.62:39711 requested a full tablet report, sending...
W20260619 12:32:27.396087 13501 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:32:27.541455 13504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:27.541862 13504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:27.541932 13504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:27.548295 13504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:27.548465 13504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:27.548585 13504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:32:27.556915 13504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45199/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43073
--tserver_master_addrs=127.0.98.62:39711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:27.558465 13504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:27.559892 13504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:27.573058 13510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:27.574000 13512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:27.575492 13509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:27.575901 13504 server_base.cc:1034] running on GCE node
I20260619 12:32:27.576426 13504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:27.577329 13504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:27.578549 13504 hybrid_clock.cc:648] HybridClock initialized: now 1781872347578508 us; error 49 us; skew 500 ppm
Jun 19 12:32:27 dist-test-slave-fg3h krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:27.581933 13504 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:32:27.583173 13504 webserver.cc:460] Webserver started at http://127.0.98.3:43073/ using document root <none> and password file <none>
I20260619 12:32:27.583952 13504 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:27.584074 13504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 12:32:27 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:32:27.611829 13504 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260619 12:32:27.616477 13520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:27.618263 13504 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260619 12:32:27.618438 13504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "19a2775f1d48400fb483448ab203a0da"
format_stamp: "Formatted at 2026-06-19 12:32:26 on dist-test-slave-fg3h"
server_key: "98b28dcd56f2c40f5d904f0b1fee88e3"
server_key_iv: "505a48ca22241b99e255ed29657e9fce"
server_key_version: "kuduclusterkey@0"
I20260619 12:32:27.619107 13504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:27.651681 13504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:27.654925 13504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:27.655136 13504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:27.655830 13504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:32:27.656807 13504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:32:27.656854 13504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:27.656915 13504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:32:27.656949 13504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:27.668030 13504 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44769
I20260619 12:32:27.668118 13640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44769 every 8 connection(s)
I20260619 12:32:27.669237 13504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:32:27.672119 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 13504
Jun 19 12:32:27 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:32:27.684113 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:27.672787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54510 (local address 127.0.98.62:39711)
0619 12:32:27.673053 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:27.673062 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:27.673277 (+ 215us) server_negotiation.cc:407] Connection header received
0619 12:32:27.673395 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:27.673398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:27.673443 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:27.673549 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:27.674318 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:27.674991 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:27.675854 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:27.676150 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:27.676258 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:27.678476 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:27.678483 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:27.678485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:27.678514 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:27.681291 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:27.681885 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:27.681888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:27.681890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:27.681998 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:27.683065 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:27.683068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:27.683070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:27.683311 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:27.683514 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:27.683806 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 12:32:27.683926 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":97,"threads_started":1}
I20260619 12:32:27.686753 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:27.671068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43571 (local address 127.0.98.62:39711)
0619 12:32:27.671196 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:27.671201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:27.672861 (+ 1660us) server_negotiation.cc:407] Connection header received
0619 12:32:27.674554 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:27.674557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:27.674601 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:27.674679 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:27.676035 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:27.676643 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:27.678504 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:27.678854 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:27.678966 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:27.681453 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:27.681458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:27.681460 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:27.681484 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:27.682947 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:27.683535 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:27.683539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:27.683541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:27.683639 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:27.684789 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:27.684793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:27.684795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:27.685011 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:27.685140 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:27.686452 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0619 12:32:27.686579 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:32:27.687613 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:27.671508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39711 (local address 127.0.98.3:43571)
0619 12:32:27.672686 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:27.672754 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:27.674326 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:27.674919 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:27.674929 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:27.675258 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:27.675898 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:27.675915 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:27.676780 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:27.676784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:27.678379 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:27.678390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:27.679018 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:27.679022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:27.679199 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:27.679765 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:27.679786 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:27.681313 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:27.683059 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:27.683065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:27.683081 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:27.683406 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:27.684309 (+ 903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:27.684312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:27.684314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:27.684499 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:27.685153 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:27.685160 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:27.686146 (+ 986us) client_negotiation.cc:769] Sending connection context
0619 12:32:27.686350 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 12:32:27.686742 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":952,"thread_start_us":143,"threads_started":1}
I20260619 12:32:27.688802 13641 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39711
I20260619 12:32:27.689036 13641 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:27.689566 13641 heartbeater.cc:511] Master 127.0.98.62:39711 requested a full tablet report, sending...
I20260619 12:32:27.690847 13168 ts_manager.cc:194] Registered new tserver with Master: 19a2775f1d48400fb483448ab203a0da (127.0.98.3:44769)
I20260619 12:32:27.692019 13168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43571
Jun 19 12:32:27 dist-test-slave-fg3h krb5kdc[12172](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781872347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:28.129931 13366 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:28.130296 13366 heartbeater.cc:511] Master 127.0.98.62:39711 requested a full tablet report, sending...
I20260619 12:32:28.131389 13168 ts_manager.cc:194] Registered new tserver with Master: c0aaf2534f754743be432fcfe04eb6b6 (127.0.98.1:44673)
I20260619 12:32:28.132208 13168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49335
Jun 19 12:32:28 dist-test-slave-fg3h krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 12:32:28.396306 13501 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:28.396621 13501 heartbeater.cc:511] Master 127.0.98.62:39711 requested a full tablet report, sending...
I20260619 12:32:28.397631 13168 ts_manager.cc:194] Registered new tserver with Master: 7060a93472474543b74b7bcc87409e16 (127.0.98.2:34499)
I20260619 12:32:28.398411 13168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44855
I20260619 12:32:28.400008 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:32:28.419564 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:28.405521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54520 (local address 127.0.98.62:39711)
0619 12:32:28.406389 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:28.406394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:28.406407 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:32:28.406629 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:28.406632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:28.406682 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:28.406758 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:28.409238 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.410165 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.413009 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.413404 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.413504 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:28.415317 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:28.415321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:28.415324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:28.415352 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:28.417195 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:28.417727 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:28.417732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:28.417734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:28.417825 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:28.418166 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:28.418170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:28.418171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:28.418395 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:28.418539 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:28.419234 (+ 695us) server_negotiation.cc:299] Negotiation successful
0619 12:32:28.419367 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":96,"threads_started":1}
I20260619 12:32:28.424127 13168 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:32:28.429752 13168 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:32:28.448549 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:28.441878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44769 (local address 127.0.0.1:35312)
0619 12:32:28.442624 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:28.442638 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:28.442714 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:28.443527 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:28.443533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:28.443558 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:32:28.443701 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:28.443711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.445163 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.445166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:28.446791 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:28.446797 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.447993 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.447997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:28.448121 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:28.448165 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 12:32:28.448268 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 12:32:28.448386 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":27,"thread_start_us":88,"threads_started":1}
I20260619 12:32:28.448699 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:28.440192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34499 (local address 127.0.0.1:36570)
0619 12:32:28.441999 (+ 1807us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:28.442012 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:28.442098 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:28.443817 (+ 1719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:28.443819 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:28.443836 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:32:28.443915 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:28.443921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.445497 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.445501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:28.446661 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:28.446671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.448303 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.448306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:28.448385 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:28.448404 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 12:32:28.448449 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 12:32:28.448500 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1733,"mutex_wait_us":732,"thread_start_us":96,"threads_started":1}
I20260619 12:32:28.449466 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:28.442296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35312 (local address 127.0.98.3:44769)
0619 12:32:28.442920 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:28.442926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:28.442947 (+ 21us) server_negotiation.cc:407] Connection header received
0619 12:32:28.443022 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:28.443037 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:28.443217 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:28.443357 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:32:28.443834 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.444985 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.446906 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.447695 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.447791 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:28.447933 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:28.449233 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0619 12:32:28.449331 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":86,"threads_started":1}
I20260619 12:32:28.450549 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:28.440421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36570 (local address 127.0.98.2:34499)
0619 12:32:28.442576 (+ 2155us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:28.442584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:28.442608 (+ 24us) server_negotiation.cc:407] Connection header received
0619 12:32:28.442690 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:28.442697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:28.442911 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:28.444060 (+ 1149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:32:28.444201 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.445377 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.447399 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.448170 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.449362 (+ 1192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:28.449538 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:28.449600 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 12:32:28.449693 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1802,"thread_start_us":111,"threads_started":1}
I20260619 12:32:28.452777 13435 tablet_service.cc:1467] Processing CreateTablet for tablet 9a8995cee4454431ab0ebae0908b9129 (DEFAULT_TABLE table=test-table [id=8090f9d36c0a42a197eb3f8553744a34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:32:28.452826 13565 tablet_service.cc:1467] Processing CreateTablet for tablet 9a8995cee4454431ab0ebae0908b9129 (DEFAULT_TABLE table=test-table [id=8090f9d36c0a42a197eb3f8553744a34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:32:28.453895 13435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a8995cee4454431ab0ebae0908b9129. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:32:28.453898 13565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a8995cee4454431ab0ebae0908b9129. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:32:28.462268 13668 tablet_bootstrap.cc:492] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da: Bootstrap starting.
I20260619 12:32:28.463547 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:28.440192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44673 (local address 127.0.0.1:54770)
0619 12:32:28.441286 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:28.441338 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:28.441461 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:28.459773 (+ 18312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:28.459776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:28.459796 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:32:28.459903 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:28.459909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.461426 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.461430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:28.462298 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:28.462307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.463123 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.463127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:28.463212 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:28.463235 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:32:28.463322 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 12:32:28.463394 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"thread_start_us":96,"threads_started":1}
I20260619 12:32:28.464594 13668 tablet_bootstrap.cc:654] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da: Neither blocks nor log segments found. Creating new log.
I20260619 12:32:28.464730 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:28.440375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54770 (local address 127.0.98.1:44673)
0619 12:32:28.459286 (+ 18911us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:28.459293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:28.459319 (+ 26us) server_negotiation.cc:407] Connection header received
0619 12:32:28.459396 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:28.459403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:28.459611 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:28.460063 (+ 452us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:32:28.460208 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.461307 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.462475 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.462990 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.463124 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:28.463269 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:28.463970 (+ 701us) server_negotiation.cc:299] Negotiation successful
0619 12:32:28.464597 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18777,"thread_start_us":99,"threads_started":1}
I20260619 12:32:28.465817 13667 tablet_bootstrap.cc:492] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16: Bootstrap starting.
I20260619 12:32:28.466789 13668 log.cc:824] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:28.467717 13282 tablet_service.cc:1467] Processing CreateTablet for tablet 9a8995cee4454431ab0ebae0908b9129 (DEFAULT_TABLE table=test-table [id=8090f9d36c0a42a197eb3f8553744a34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:32:28.468762 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a8995cee4454431ab0ebae0908b9129. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:32:28.469024 13667 tablet_bootstrap.cc:654] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16: Neither blocks nor log segments found. Creating new log.
I20260619 12:32:28.469841 13667 log.cc:824] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:28.472314 13668 tablet_bootstrap.cc:492] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da: No bootstrap required, opened a new log
I20260619 12:32:28.472503 13668 ts_tablet_manager.cc:1397] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 12:32:28.476924 13668 raft_consensus.cc:348] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7060a93472474543b74b7bcc87409e16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34499 } } peers { permanent_uuid: "c0aaf2534f754743be432fcfe04eb6b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44673 } } peers { permanent_uuid: "19a2775f1d48400fb483448ab203a0da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44769 } }
I20260619 12:32:28.477453 13668 raft_consensus.cc:374] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:32:28.477553 13668 raft_consensus.cc:729] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a2775f1d48400fb483448ab203a0da, State: Initialized, Role: FOLLOWER
I20260619 12:32:28.478440 13668 consensus_queue.cc:260] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7060a93472474543b74b7bcc87409e16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34499 } } peers { permanent_uuid: "c0aaf2534f754743be432fcfe04eb6b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44673 } } peers { permanent_uuid: "19a2775f1d48400fb483448ab203a0da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44769 } }
I20260619 12:32:28.479528 13668 ts_tablet_manager.cc:1428] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 12:32:28.479943 13641 heartbeater.cc:503] Master 127.0.98.62:39711 was elected leader, sending a full tablet report...
I20260619 12:32:28.480324 13671 tablet_bootstrap.cc:492] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6: Bootstrap starting.
I20260619 12:32:28.482754 13671 tablet_bootstrap.cc:654] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6: Neither blocks nor log segments found. Creating new log.
I20260619 12:32:28.483664 13671 log.cc:824] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:28.484846 13667 tablet_bootstrap.cc:492] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16: No bootstrap required, opened a new log
I20260619 12:32:28.485124 13667 ts_tablet_manager.cc:1397] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20260619 12:32:28.485728 13671 tablet_bootstrap.cc:492] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6: No bootstrap required, opened a new log
I20260619 12:32:28.485901 13671 ts_tablet_manager.cc:1397] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260619 12:32:28.488430 13667 raft_consensus.cc:348] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7060a93472474543b74b7bcc87409e16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34499 } } peers { permanent_uuid: "c0aaf2534f754743be432fcfe04eb6b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44673 } } peers { permanent_uuid: "19a2775f1d48400fb483448ab203a0da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44769 } }
I20260619 12:32:28.488674 13667 raft_consensus.cc:374] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:32:28.488734 13667 raft_consensus.cc:729] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7060a93472474543b74b7bcc87409e16, State: Initialized, Role: FOLLOWER
I20260619 12:32:28.489370 13667 consensus_queue.cc:260] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7060a93472474543b74b7bcc87409e16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34499 } } peers { permanent_uuid: "c0aaf2534f754743be432fcfe04eb6b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44673 } } peers { permanent_uuid: "19a2775f1d48400fb483448ab203a0da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44769 } }
I20260619 12:32:28.490526 13667 ts_tablet_manager.cc:1428] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:32:28.490689 13671 raft_consensus.cc:348] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7060a93472474543b74b7bcc87409e16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34499 } } peers { permanent_uuid: "c0aaf2534f754743be432fcfe04eb6b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44673 } } peers { permanent_uuid: "19a2775f1d48400fb483448ab203a0da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44769 } }
I20260619 12:32:28.490952 13501 heartbeater.cc:503] Master 127.0.98.62:39711 was elected leader, sending a full tablet report...
I20260619 12:32:28.490989 13671 raft_consensus.cc:374] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:32:28.491046 13671 raft_consensus.cc:729] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0aaf2534f754743be432fcfe04eb6b6, State: Initialized, Role: FOLLOWER
I20260619 12:32:28.491469 13671 consensus_queue.cc:260] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7060a93472474543b74b7bcc87409e16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34499 } } peers { permanent_uuid: "c0aaf2534f754743be432fcfe04eb6b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44673 } } peers { permanent_uuid: "19a2775f1d48400fb483448ab203a0da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44769 } }
I20260619 12:32:28.501616 13671 ts_tablet_manager.cc:1428] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6: Time spent starting tablet: real 0.016s user 0.003s sys 0.003s
I20260619 12:32:28.502344 13366 heartbeater.cc:503] Master 127.0.98.62:39711 was elected leader, sending a full tablet report...
I20260619 12:32:28.509720 13675 raft_consensus.cc:482] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:32:28.509989 13675 raft_consensus.cc:504] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7060a93472474543b74b7bcc87409e16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34499 } } peers { permanent_uuid: "c0aaf2534f754743be432fcfe04eb6b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44673 } } peers { permanent_uuid: "19a2775f1d48400fb483448ab203a0da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44769 } }
I20260619 12:32:28.511432 13675 leader_election.cc:288] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7060a93472474543b74b7bcc87409e16 (127.0.98.2:34499), 19a2775f1d48400fb483448ab203a0da (127.0.98.3:44769)
I20260619 12:32:28.516889 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:28.511631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34499 (local address 127.0.98.1:40617)
0619 12:32:28.512363 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:28.512376 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:28.512505 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:28.512842 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:28.512845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:28.512860 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:32:28.512995 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:28.513000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.514057 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.514060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:28.515398 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:28.515409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.516423 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.516426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:28.516548 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:28.516571 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:32:28.516642 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 12:32:28.516708 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":60,"thread_start_us":131,"threads_started":1}
I20260619 12:32:28.517030 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:28.512030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60835 (local address 127.0.98.3:44769)
0619 12:32:28.512140 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:28.512145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:28.512159 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:32:28.512218 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:28.512221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:28.512270 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:28.512350 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:32:28.513398 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.514157 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.515681 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.516280 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.516362 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:28.516423 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:28.516834 (+ 411us) server_negotiation.cc:299] Negotiation successful
0619 12:32:28.516912 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 12:32:28.517329 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:28.512062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40617 (local address 127.0.98.2:34499)
0619 12:32:28.512183 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:28.512188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:28.512444 (+ 256us) server_negotiation.cc:407] Connection header received
0619 12:32:28.512614 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:28.512617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:28.512669 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:28.512717 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:32:28.513143 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.513921 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.515556 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.516180 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.516261 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:28.516327 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:28.517136 (+ 809us) server_negotiation.cc:299] Negotiation successful
0619 12:32:28.517217 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 12:32:28.517920 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:28.511631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44769 (local address 127.0.98.1:60835)
0619 12:32:28.511998 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:28.512014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:28.512129 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:28.513143 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:28.513146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:28.513160 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:32:28.513264 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:28.513269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.514305 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.514308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:28.515517 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:28.515528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.516516 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.516519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:28.516616 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:28.516717 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 12:32:28.517696 (+ 979us) client_negotiation.cc:240] Negotiation successful
0619 12:32:28.517775 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":139,"threads_started":1}
I20260619 12:32:28.517900 13452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8995cee4454431ab0ebae0908b9129" candidate_uuid: "c0aaf2534f754743be432fcfe04eb6b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7060a93472474543b74b7bcc87409e16" is_pre_election: true
I20260619 12:32:28.518208 13452 raft_consensus.cc:2454] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0aaf2534f754743be432fcfe04eb6b6 in term 0.
I20260619 12:32:28.518707 13254 leader_election.cc:302] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7060a93472474543b74b7bcc87409e16, c0aaf2534f754743be432fcfe04eb6b6; no voters:
I20260619 12:32:28.519003 13675 raft_consensus.cc:2781] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:32:28.519079 13675 raft_consensus.cc:482] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:32:28.519124 13675 raft_consensus.cc:3037] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:32:28.519071 13595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8995cee4454431ab0ebae0908b9129" candidate_uuid: "c0aaf2534f754743be432fcfe04eb6b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a2775f1d48400fb483448ab203a0da" is_pre_election: true
I20260619 12:32:28.519349 13595 raft_consensus.cc:2454] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0aaf2534f754743be432fcfe04eb6b6 in term 0.
I20260619 12:32:28.520295 13675 raft_consensus.cc:504] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7060a93472474543b74b7bcc87409e16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34499 } } peers { permanent_uuid: "c0aaf2534f754743be432fcfe04eb6b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44673 } } peers { permanent_uuid: "19a2775f1d48400fb483448ab203a0da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44769 } }
I20260619 12:32:28.520697 13675 leader_election.cc:288] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 [CANDIDATE]: Term 1 election: Requested vote from peers 7060a93472474543b74b7bcc87409e16 (127.0.98.2:34499), 19a2775f1d48400fb483448ab203a0da (127.0.98.3:44769)
I20260619 12:32:28.521030 13452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8995cee4454431ab0ebae0908b9129" candidate_uuid: "c0aaf2534f754743be432fcfe04eb6b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7060a93472474543b74b7bcc87409e16"
I20260619 12:32:28.521194 13452 raft_consensus.cc:3037] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:32:28.521234 13595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8995cee4454431ab0ebae0908b9129" candidate_uuid: "c0aaf2534f754743be432fcfe04eb6b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a2775f1d48400fb483448ab203a0da"
I20260619 12:32:28.521347 13595 raft_consensus.cc:3037] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:32:28.522528 13452 raft_consensus.cc:2454] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0aaf2534f754743be432fcfe04eb6b6 in term 1.
I20260619 12:32:28.522688 13595 raft_consensus.cc:2454] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0aaf2534f754743be432fcfe04eb6b6 in term 1.
I20260619 12:32:28.523080 13255 leader_election.cc:302] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a2775f1d48400fb483448ab203a0da, c0aaf2534f754743be432fcfe04eb6b6; no voters:
I20260619 12:32:28.523244 13675 raft_consensus.cc:2781] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:32:28.523527 13675 raft_consensus.cc:686] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 [term 1 LEADER]: Becoming Leader. State: Replica: c0aaf2534f754743be432fcfe04eb6b6, State: Running, Role: LEADER
I20260619 12:32:28.523885 13675 consensus_queue.cc:237] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7060a93472474543b74b7bcc87409e16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34499 } } peers { permanent_uuid: "c0aaf2534f754743be432fcfe04eb6b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44673 } } peers { permanent_uuid: "19a2775f1d48400fb483448ab203a0da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44769 } }
I20260619 12:32:28.527262 13168 catalog_manager.cc:5495] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0aaf2534f754743be432fcfe04eb6b6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c0aaf2534f754743be432fcfe04eb6b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7060a93472474543b74b7bcc87409e16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0aaf2534f754743be432fcfe04eb6b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a2775f1d48400fb483448ab203a0da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44769 } health_report { overall_health: UNKNOWN } } }
I20260619 12:32:28.548996 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:28.544313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54774 (local address 127.0.98.1:44673)
0619 12:32:28.544430 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:28.544435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:28.544501 (+ 66us) server_negotiation.cc:407] Connection header received
0619 12:32:28.544697 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:28.544700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:28.544770 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:28.544880 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:32:28.545213 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.545979 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.547240 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:28.547566 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:28.547670 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:28.548012 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:32:28.548568 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:32:28.548682 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:28.548780 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 12:32:28.548841 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 12:32:28.556311 13595 raft_consensus.cc:1261] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da [term 1 FOLLOWER]: Refusing update from remote peer c0aaf2534f754743be432fcfe04eb6b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:32:28.556314 13452 raft_consensus.cc:1261] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16 [term 1 FOLLOWER]: Refusing update from remote peer c0aaf2534f754743be432fcfe04eb6b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:32:28.557125 13675 consensus_queue.cc:1035] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7060a93472474543b74b7bcc87409e16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:32:28.557665 13678 consensus_queue.cc:1035] T 9a8995cee4454431ab0ebae0908b9129 P c0aaf2534f754743be432fcfe04eb6b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a2775f1d48400fb483448ab203a0da" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:32:28.571151 13682 mvcc.cc:205] Tried to move back new op lower bound from 7298549139678584832 to 7298549139556515840. Current Snapshot: MvccSnapshot[applied={T|T < 7298549139678584832}]
I20260619 12:32:28.574220 13683 mvcc.cc:205] Tried to move back new op lower bound from 7298549139678584832 to 7298549139556515840. Current Snapshot: MvccSnapshot[applied={T|T < 7298549139678584832}]
I20260619 12:32:28.583107 13168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:32:28.583300 13168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:32:28.586356 13168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1aa47aac4dde42f49fb90523119d52f3: Sending DeleteTablet for 3 replicas of tablet 9a8995cee4454431ab0ebae0908b9129
I20260619 12:32:28.587656 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a8995cee4454431ab0ebae0908b9129 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:32:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54770
I20260619 12:32:28.588210 13565 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a8995cee4454431ab0ebae0908b9129 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:32:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35312
I20260619 12:32:28.589262 13435 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a8995cee4454431ab0ebae0908b9129 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:32:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36570
I20260619 12:32:28.590709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 13206
I20260619 12:32:28.591563 13692 tablet_replica.cc:330] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da: stopping tablet replica
I20260619 12:32:28.591894 13692 raft_consensus.cc:2229] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:32:28.592216 13692 raft_consensus.cc:2258] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:32:28.593427 13693 tablet_replica.cc:330] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16: stopping tablet replica
I20260619 12:32:28.593720 13693 raft_consensus.cc:2229] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:32:28.593984 13693 raft_consensus.cc:2258] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:32:28.594838 13692 ts_tablet_manager.cc:1905] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:32:28.597489 13692 ts_tablet_manager.cc:1918] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:32:28.597595 13692 log.cc:1196] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/9a8995cee4454431ab0ebae0908b9129
I20260619 12:32:28.597879 13692 ts_tablet_manager.cc:1939] T 9a8995cee4454431ab0ebae0908b9129 P 19a2775f1d48400fb483448ab203a0da: Deleting consensus metadata
I20260619 12:32:28.599421 13154 catalog_manager.cc:4841] TS 19a2775f1d48400fb483448ab203a0da (127.0.98.3:44769): tablet 9a8995cee4454431ab0ebae0908b9129 (table test-table [id=8090f9d36c0a42a197eb3f8553744a34]) successfully deleted
I20260619 12:32:28.600364 13693 ts_tablet_manager.cc:1905] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 12:32:28.602727 13155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44673 (error 108)
I20260619 12:32:28.603061 13693 ts_tablet_manager.cc:1918] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:32:28.603142 13693 log.cc:1196] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/9a8995cee4454431ab0ebae0908b9129
I20260619 12:32:28.603425 13693 ts_tablet_manager.cc:1939] T 9a8995cee4454431ab0ebae0908b9129 P 7060a93472474543b74b7bcc87409e16: Deleting consensus metadata
I20260619 12:32:28.604249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 13370
I20260619 12:32:28.604279 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:28.603989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44673 (local address 127.0.0.1:54778)
0619 12:32:28.604122 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:28.604196 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 12:32:28.604434 13155 catalog_manager.cc:4584] TS c0aaf2534f754743be432fcfe04eb6b6 (127.0.98.1:44673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a8995cee4454431ab0ebae0908b9129: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44673: connect: Connection refused (error 111)
I20260619 12:32:28.611555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 13504
I20260619 12:32:28.615090 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:28.614715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34499 (local address 127.0.0.1:36576)
0619 12:32:28.614917 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:28.614995 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260619 12:32:28.618340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 13135
I20260619 12:32:28.624544 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 12:32:28.959448 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 12:32:28.959550 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 12:32:29.295436 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 12:32:29.295 UTC [12202] LOG: received smart shutdown request
2026-06-19 12:32:29.299 UTC [12202] LOG: background worker "logical replication launcher" (PID 12211) exited with exit code 1
2026-06-19 12:32:29.299 UTC [12206] LOG: shutting down
2026-06-19 12:32:29.453 UTC [12202] LOG: database system is shut down
2026-06-19T12:32:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49385 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:32:29 dist-test-slave-fg3h krb5kdc[13703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:32:29 dist-test-slave-fg3h krb5kdc[13703](info): set up 2 sockets
Jun 19 12:32:29 dist-test-slave-fg3h krb5kdc[13703](info): commencing operation
krb5kdc: starting...
W20260619 12:32:30.625240 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:32:30 dist-test-slave-fg3h krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:32:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:32:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:32:30.746156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42863
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42863,127.0.98.61:45009,127.0.98.60:32835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:32:30.858244 13716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:30.858529 13716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:30.858572 13716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:30.863245 13716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:32:30.863335 13716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:30.863359 13716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:30.863432 13716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:32:30.863467 13716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:32:30.869599 13716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42863,127.0.98.61:45009,127.0.98.60:32835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:30.870682 13716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:30.871752 13716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:30.880286 13721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:30.880378 13724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:30.880406 13722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:30.880999 13716 server_base.cc:1034] running on GCE node
I20260619 12:32:30.881671 13716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:30.882903 13716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:30.884099 13716 hybrid_clock.cc:648] HybridClock initialized: now 1781872350884062 us; error 49 us; skew 500 ppm
Jun 19 12:32:30 dist-test-slave-fg3h krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:30.887146 13716 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:32:30.888128 13716 webserver.cc:460] Webserver started at http://127.0.98.62:45677/ using document root <none> and password file <none>
I20260619 12:32:30.888767 13716 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:30.888866 13716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:30.889405 13716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:32:30.891616 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc"
format_stamp: "Formatted at 2026-06-19 12:32:30 on dist-test-slave-fg3h"
server_key: "1f759d8210bbb458a50f03f9ab13938d"
server_key_iv: "053a9dadab3c3a08f3628a6e86e5450f"
server_key_version: "encryptionkey@0"
I20260619 12:32:30.892299 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc"
format_stamp: "Formatted at 2026-06-19 12:32:30 on dist-test-slave-fg3h"
server_key: "1f759d8210bbb458a50f03f9ab13938d"
server_key_iv: "053a9dadab3c3a08f3628a6e86e5450f"
server_key_version: "encryptionkey@0"
I20260619 12:32:30.896359 13716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:32:30.898986 13731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:30.900295 13716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 12:32:30.900449 13716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc"
format_stamp: "Formatted at 2026-06-19 12:32:30 on dist-test-slave-fg3h"
server_key: "1f759d8210bbb458a50f03f9ab13938d"
server_key_iv: "053a9dadab3c3a08f3628a6e86e5450f"
server_key_version: "encryptionkey@0"
I20260619 12:32:30.900569 13716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:30.921468 13716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:30.924489 13716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:30.924701 13716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:30.932073 13716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42863
I20260619 12:32:30.932086 13783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42863 every 8 connection(s)
I20260619 12:32:30.933128 13716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:32:30.935922 13784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:32:30.939738 13784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:32:30.942679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 13716
I20260619 12:32:30.942771 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:32:30.943099 392 external_mini_cluster.cc:1428] Setting key 355fb7a83a919e728f2529d38139b9a7
Jun 19 12:32:30 dist-test-slave-fg3h krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260619 12:32:30.957135 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:30.942062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55448 (local address 127.0.98.62:42863)
0619 12:32:30.942569 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:30.942590 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:30.943325 (+ 735us) server_negotiation.cc:407] Connection header received
0619 12:32:30.944845 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:30.944852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:30.945123 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:30.945329 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:30.946092 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:30.947589 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:30.949160 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:30.949540 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:30.949659 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:30.952317 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:30.952322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:30.952332 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:30.952351 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:30.953999 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:30.954483 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:30.954488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:30.954490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:30.954595 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:30.954934 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:30.954937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:30.954939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:30.955308 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:30.955446 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:30.955815 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 12:32:30.956095 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":347,"spinlock_wait_cycles":31616,"thread_start_us":156,"threads_started":1}
I20260619 12:32:30.957135 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:30.942062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42863 (local address 127.0.0.1:55448)
0619 12:32:30.943158 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:30.943213 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:30.944478 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:30.945345 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:30.945349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:30.945692 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:30.945941 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:30.945953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:30.947742 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:30.947746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:30.948974 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:30.949002 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:30.949713 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:30.949718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:30.949858 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:30.950495 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:30.950522 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:30.952160 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:30.954131 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:30.954141 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:30.954151 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:30.954371 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:30.954691 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:30.954695 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:30.954697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:30.954849 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:30.955428 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:30.955435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:30.955567 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 12:32:30.955757 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 12:32:30.956142 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":109,"thread_start_us":147,"threads_started":1}
I20260619 12:32:30.959699 13784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:32:30.960457 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:30.960195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45009 (local address 127.0.0.1:52290)
0619 12:32:30.960326 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:30.960399 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 12:32:30.960728 13735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45009: connect: Connection refused (error 111)
I20260619 12:32:30.961673 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:30.961474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45009 (local address 127.0.0.1:52292)
0619 12:32:30.961577 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:30.961620 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 12:32:30.961898 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45009: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45009: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 12:32:30.972440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45009
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42863,127.0.98.61:45009,127.0.98.60:32835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:32:31.011142 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } attempt: 1
I20260619 12:32:31.012034 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.011735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45009 (local address 127.0.0.1:52308)
0619 12:32:31.011890 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.011954 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 12:32:31.012769 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.012543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45009 (local address 127.0.0.1:52320)
0619 12:32:31.012652 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.012708 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 12:32:31.012904 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45009: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45009: connect: Connection refused (error 111)
I20260619 12:32:31.081161 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } attempt: 2
I20260619 12:32:31.081971 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.081672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45009 (local address 127.0.0.1:52334)
0619 12:32:31.081828 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.081894 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 12:32:31.082690 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.082500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45009 (local address 127.0.0.1:52342)
0619 12:32:31.082595 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.082639 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 12:32:31.082832 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45009: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45009: connect: Connection refused (error 111)
W20260619 12:32:31.086706 13791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:31.086971 13791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:31.087047 13791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:31.091248 13791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:32:31.091305 13791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:31.091324 13791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:31.091341 13791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:32:31.091391 13791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:32:31.096922 13791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42863,127.0.98.61:45009,127.0.98.60:32835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:31.098049 13791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:31.099092 13791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:31.108057 13797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:31.108150 13799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:31.108002 13796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:31.108363 13791 server_base.cc:1034] running on GCE node
I20260619 12:32:31.108994 13791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:31.110251 13791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:31.111451 13791 hybrid_clock.cc:648] HybridClock initialized: now 1781872351111410 us; error 57 us; skew 500 ppm
Jun 19 12:32:31 dist-test-slave-fg3h krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:31.114516 13791 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 12:32:31.115473 13791 webserver.cc:460] Webserver started at http://127.0.98.61:43037/ using document root <none> and password file <none>
I20260619 12:32:31.116052 13791 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:31.116146 13791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:31.116385 13791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:32:31.118532 13791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-1/data/instance:
uuid: "e1842df1675546c888091f5042a91a80"
format_stamp: "Formatted at 2026-06-19 12:32:31 on dist-test-slave-fg3h"
server_key: "615180107b9cbd4d473d36ce158932dc"
server_key_iv: "cb413e531d47ab257857febd79ef5061"
server_key_version: "encryptionkey@0"
I20260619 12:32:31.119124 13791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-1/wal/instance:
uuid: "e1842df1675546c888091f5042a91a80"
format_stamp: "Formatted at 2026-06-19 12:32:31 on dist-test-slave-fg3h"
server_key: "615180107b9cbd4d473d36ce158932dc"
server_key_iv: "cb413e531d47ab257857febd79ef5061"
server_key_version: "encryptionkey@0"
I20260619 12:32:31.123428 13791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:32:31.126175 13806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:31.127458 13791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:32:31.127612 13791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-1/wal
uuid: "e1842df1675546c888091f5042a91a80"
format_stamp: "Formatted at 2026-06-19 12:32:31 on dist-test-slave-fg3h"
server_key: "615180107b9cbd4d473d36ce158932dc"
server_key_iv: "cb413e531d47ab257857febd79ef5061"
server_key_version: "encryptionkey@0"
I20260619 12:32:31.127735 13791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:31.146659 13791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:31.150069 13791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:31.150286 13791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:31.157637 13791 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45009
I20260619 12:32:31.157661 13858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45009 every 8 connection(s)
I20260619 12:32:31.158602 13791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-1/data/info.pb
I20260619 12:32:31.159125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 13791
I20260619 12:32:31.159246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-1/wal/instance
I20260619 12:32:31.159533 392 external_mini_cluster.cc:1428] Setting key 4b7baa3a51b697676d171ce43fa318f6
I20260619 12:32:31.161968 13859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:32:31.167502 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260619 12:32:31.174022 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } attempt: 3
Jun 19 12:32:31 dist-test-slave-fg3h krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 12:32:31 dist-test-slave-fg3h krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 12:32:31.185290 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.174512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45009 (local address 127.0.0.1:52348)
0619 12:32:31.174704 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.174723 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:31.174808 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:31.175470 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:31.175473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:31.175676 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:31.175778 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:31.175784 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.176898 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.176902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:31.177711 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:31.177720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.178332 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.178336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:31.178731 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.179201 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:31.179212 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:31.180847 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:31.183124 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:31.183128 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:31.183130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:31.183396 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:31.183790 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:31.183793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:31.183796 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:31.183882 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:31.184639 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:31.184642 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:31.184773 (+ 131us) client_negotiation.cc:769] Sending connection context
0619 12:32:31.184880 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 12:32:31.185020 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260619 12:32:31.185627 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.169865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.0.98.62:42863)
0619 12:32:31.170096 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:31.170103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:31.171062 (+ 959us) server_negotiation.cc:407] Connection header received
0619 12:32:31.172533 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:31.172538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:31.172604 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:31.172711 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:31.173975 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.174975 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.177258 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.177609 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.177715 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.180955 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:31.180958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:31.180960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:31.180984 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:31.183511 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.184124 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.184130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.184139 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.184212 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.184534 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.184537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.184539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.184753 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:31.184850 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:31.185343 (+ 493us) server_negotiation.cc:299] Negotiation successful
0619 12:32:31.185467 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 12:32:31.186544 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.174614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52348 (local address 127.0.98.61:45009)
0619 12:32:31.175042 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:31.175057 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:31.175085 (+ 28us) server_negotiation.cc:407] Connection header received
0619 12:32:31.175134 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:31.175141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:31.175318 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:31.175483 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:31.175894 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.176772 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.177847 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.178209 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.178328 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.181000 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:31.181005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:31.181016 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:31.181038 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:31.182988 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.183508 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.183514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.183531 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.183683 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.183983 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.183985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.183986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.184442 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:31.184791 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:31.184999 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 12:32:31.185604 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":130,"threads_started":1}
I20260619 12:32:31.186548 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.170287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42863 (local address 127.0.0.1:55454)
0619 12:32:31.170882 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.170943 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:31.172246 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:31.172884 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:31.172891 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:31.173293 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:31.173792 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:31.173806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.175251 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.175259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:31.177034 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:31.177052 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.177815 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.177820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:31.178084 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.178791 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:31.178818 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:31.180773 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:31.183684 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:31.183693 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:31.183699 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:31.184009 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:31.184307 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:31.184311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:31.184313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:31.184433 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:31.184907 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:31.184914 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:31.185038 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 12:32:31.185789 (+ 751us) client_negotiation.cc:240] Negotiation successful
0619 12:32:31.186001 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 12:32:31.188015 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:32:31.188768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32835
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42863,127.0.98.61:45009,127.0.98.60:32835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:32:31.188814 13784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:32:31.189603 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.189263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32835 (local address 127.0.0.1:46148)
0619 12:32:31.189418 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.189506 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 12:32:31.190289 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.190050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32835 (local address 127.0.0.1:46158)
0619 12:32:31.190187 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.190240 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260619 12:32:31.190480 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32835: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32835: connect: Connection refused (error 111)
Jun 19 12:32:31 dist-test-slave-fg3h krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 12:32:31.199477 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.188540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45009 (local address 127.0.0.1:52358)
0619 12:32:31.190428 (+ 1888us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.190446 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:31.190560 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:31.190862 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:31.190865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:31.191004 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:31.191093 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:31.191098 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.192285 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.192287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:31.192849 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:31.192856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.193694 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.193697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:31.193792 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.194330 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:31.194341 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:31.195579 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:31.197907 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:31.197910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:31.197912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:31.198102 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:31.198442 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:31.198444 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:31.198445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:31.198562 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:31.199069 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:31.199071 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:31.199146 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 12:32:31.199231 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 12:32:31.199335 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1796,"mutex_wait_us":1}
I20260619 12:32:31.199733 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.188572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52358 (local address 127.0.98.61:45009)
0619 12:32:31.188879 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:31.188883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:31.190647 (+ 1764us) server_negotiation.cc:407] Connection header received
0619 12:32:31.190680 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:31.190684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:31.190729 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:31.190808 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:31.191221 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.192162 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.192986 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.193569 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.193748 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.195723 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:31.195731 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:31.195738 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:31.195772 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:31.197786 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.198209 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.198216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.198220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.198325 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.198667 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.198673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.198675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.198952 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:31.199075 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:31.199331 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 12:32:31.199478 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":238}
I20260619 12:32:31.200444 13859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:32:31.201207 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.200902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32835 (local address 127.0.0.1:46162)
0619 12:32:31.201017 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.201144 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 12:32:31.201747 13808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32835: connect: Connection refused (error 111)
I20260619 12:32:31.202558 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.202344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32835 (local address 127.0.0.1:46164)
0619 12:32:31.202452 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.202496 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 12:32:31.202721 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32835: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32835: connect: Connection refused (error 111)
I20260619 12:32:31.221644 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } attempt: 1
I20260619 12:32:31.222546 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.222223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32835 (local address 127.0.0.1:46172)
0619 12:32:31.222393 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.222453 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 12:32:31.223313 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.223095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32835 (local address 127.0.0.1:46186)
0619 12:32:31.223208 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.223259 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 12:32:31.223569 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32835: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32835: connect: Connection refused (error 111)
I20260619 12:32:31.251977 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } attempt: 1
I20260619 12:32:31.253017 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.252705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32835 (local address 127.0.0.1:46188)
0619 12:32:31.252871 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.252937 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 12:32:31.253793 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.253537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32835 (local address 127.0.0.1:46204)
0619 12:32:31.253678 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.253739 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260619 12:32:31.254010 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32835: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32835: connect: Connection refused (error 111)
I20260619 12:32:31.298753 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } attempt: 2
I20260619 12:32:31.299809 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.299448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32835 (local address 127.0.0.1:46216)
0619 12:32:31.299631 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.299699 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 12:32:31.300586 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.300363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32835 (local address 127.0.0.1:46220)
0619 12:32:31.300478 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.300531 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 12:32:31.300724 13784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32835: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32835: connect: Connection refused (error 111)
W20260619 12:32:31.310078 13865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:31.310371 13865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:31.310477 13865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:31.314594 13865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:32:31.314682 13865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:31.314721 13865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:31.314779 13865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:32:31.314821 13865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:32:31.320124 13865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42863,127.0.98.61:45009,127.0.98.60:32835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:31.321259 13865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:31.322176 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } attempt: 2
I20260619 12:32:31.322363 13865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:32:31.322984 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.322660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32835 (local address 127.0.0.1:46224)
0619 12:32:31.322839 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.322909 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 12:32:31.323676 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.323473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32835 (local address 127.0.0.1:46236)
0619 12:32:31.323577 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.323628 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 12:32:31.323833 13859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32835: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32835: connect: Connection refused (error 111)
W20260619 12:32:31.331125 13871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:31.331143 13874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:31.331147 13872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:31.332187 13865 server_base.cc:1034] running on GCE node
I20260619 12:32:31.332733 13865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:31.333877 13865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:31.335042 13865 hybrid_clock.cc:648] HybridClock initialized: now 1781872351335012 us; error 42 us; skew 500 ppm
Jun 19 12:32:31 dist-test-slave-fg3h krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:31.338006 13865 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 12:32:31.338887 13865 webserver.cc:460] Webserver started at http://127.0.98.60:42593/ using document root <none> and password file <none>
I20260619 12:32:31.339450 13865 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:31.339511 13865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:31.339744 13865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:32:31.341862 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-2/data/instance:
uuid: "aad8dbfb8c8c413eb51cf86b80d72ead"
format_stamp: "Formatted at 2026-06-19 12:32:31 on dist-test-slave-fg3h"
server_key: "06170e83762c990e4debbcd0738893ff"
server_key_iv: "00d4a51077f2de204b5b1955e7022217"
server_key_version: "encryptionkey@0"
I20260619 12:32:31.342432 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-2/wal/instance:
uuid: "aad8dbfb8c8c413eb51cf86b80d72ead"
format_stamp: "Formatted at 2026-06-19 12:32:31 on dist-test-slave-fg3h"
server_key: "06170e83762c990e4debbcd0738893ff"
server_key_iv: "00d4a51077f2de204b5b1955e7022217"
server_key_version: "encryptionkey@0"
I20260619 12:32:31.346203 13865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:32:31.348503 13881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:31.349877 13865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:32:31.350003 13865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-2/wal
uuid: "aad8dbfb8c8c413eb51cf86b80d72ead"
format_stamp: "Formatted at 2026-06-19 12:32:31 on dist-test-slave-fg3h"
server_key: "06170e83762c990e4debbcd0738893ff"
server_key_iv: "00d4a51077f2de204b5b1955e7022217"
server_key_version: "encryptionkey@0"
I20260619 12:32:31.350117 13865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:31.361070 13865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:31.364488 13865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:31.364691 13865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:31.371680 13865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:32835
I20260619 12:32:31.371786 13933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:32835 every 8 connection(s)
I20260619 12:32:31.372678 13865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-2/data/info.pb
I20260619 12:32:31.375376 13934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:32:31.376052 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 13865
I20260619 12:32:31.376139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781872188319340-392-0/minicluster-data/master-2/wal/instance
I20260619 12:32:31.376443 392 external_mini_cluster.cc:1428] Setting key 2c3d24a95c06b32467c196fa59a2b9d5
I20260619 12:32:31.379873 13934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:32:31 dist-test-slave-fg3h krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 12:32:31 dist-test-slave-fg3h krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:32:31.391759 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.381750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55456 (local address 127.0.98.62:42863)
0619 12:32:31.381908 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:31.381915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:31.382294 (+ 379us) server_negotiation.cc:407] Connection header received
0619 12:32:31.382491 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:31.382494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:31.382545 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:31.382632 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:31.383333 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.383980 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.385007 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.385433 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.385516 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.388324 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:31.388327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:31.388329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:31.388353 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:31.389926 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.390407 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.390409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.390410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.390531 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.390858 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.390861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.390862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.391068 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:31.391165 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:31.391495 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 12:32:31.391620 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 12:32:31.391917 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.377852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46246 (local address 127.0.98.60:32835)
0619 12:32:31.378285 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:31.378303 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:31.378348 (+ 45us) server_negotiation.cc:407] Connection header received
0619 12:32:31.379148 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:31.379170 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:31.379588 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:31.379948 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:31.380832 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.382153 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.383044 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.383617 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.383736 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.387118 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:31.387128 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:31.387145 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:31.387171 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:31.388874 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.389373 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.389378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.389382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.389503 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.389794 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.389797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.389799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.390164 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:31.390313 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:31.390676 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 12:32:31.390965 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":94,"threads_started":1}
I20260619 12:32:31.391954 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.381728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42863 (local address 127.0.0.1:55456)
0619 12:32:31.382163 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.382224 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:31.382379 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:31.382677 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:31.382682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:31.383074 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:31.383207 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:31.383214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.384113 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.384117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:31.384840 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:31.384849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.385593 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.385597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:31.385711 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.386313 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:31.386332 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:31.388157 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:31.390029 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:31.390038 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:31.390044 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:31.390316 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:31.390644 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:31.390648 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:31.390650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:31.390755 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:31.391170 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:31.391174 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:31.391281 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 12:32:31.391392 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 12:32:31.391516 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":181,"threads_started":1}
I20260619 12:32:31.393240 13934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:32:31 dist-test-slave-fg3h krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 12:32:31 dist-test-slave-fg3h krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:32:31.399890 13784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } attempt: 3
I20260619 12:32:31.401976 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.393762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45009 (local address 127.0.0.1:52364)
0619 12:32:31.393925 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.393939 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:31.394005 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:31.394234 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:31.394237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:31.394393 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:31.394810 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:31.394816 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.395540 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.395546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:31.396039 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:31.396046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.396520 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.396523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:31.396630 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.397139 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:31.397157 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:31.398440 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:31.400177 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:31.400181 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:31.400185 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:31.400444 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:31.400811 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:31.400814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:31.400816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:31.400899 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:31.401362 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:31.401366 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:31.401461 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 12:32:31.401688 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 12:32:31.401819 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 12:32:31.401985 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.393855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52364 (local address 127.0.98.61:45009)
0619 12:32:31.394004 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:31.394010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:31.394028 (+ 18us) server_negotiation.cc:407] Connection header received
0619 12:32:31.394091 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:31.394094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:31.394131 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:31.394221 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:31.394933 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.395434 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.396158 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.396411 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.396520 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.398581 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:31.398585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:31.398587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:31.398617 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:31.400070 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.400600 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.400603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.400605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.400693 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.400981 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.400987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.400989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.401250 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:31.401360 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:31.401770 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 12:32:31.401876 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 12:32:31.403035 13934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:32:31.403185 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.394268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55460 (local address 127.0.98.62:42863)
0619 12:32:31.394370 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:31.394374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:31.394442 (+ 68us) server_negotiation.cc:407] Connection header received
0619 12:32:31.394571 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:31.394574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:31.394612 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:31.394710 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:31.395264 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.395800 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.396541 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.396772 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.396878 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.399237 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:31.399243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:31.399250 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:31.399271 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:31.401113 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.401535 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.401539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.401541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.401631 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.402353 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.402357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.402358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.402580 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:31.402804 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:31.402939 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 12:32:31.403049 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 12:32:31 dist-test-slave-fg3h krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 12:32:31 dist-test-slave-fg3h krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 12:32:31 dist-test-slave-fg3h krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 12:32:31.410665 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.400568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32835 (local address 127.0.0.1:46252)
0619 12:32:31.400732 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.400746 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:31.400821 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:31.401047 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:31.401050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:31.401249 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:31.401353 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:31.401360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.402494 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.402497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:31.403301 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:31.403313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.404182 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.404189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:31.404273 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.404724 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:31.404739 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:31.406080 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:31.408924 (+ 2844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:31.408928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:31.408930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:31.409193 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:31.409541 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:31.409546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:31.409548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:31.409645 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:31.410129 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:31.410132 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:31.410247 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 12:32:31.410359 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 12:32:31.410491 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 12:32:31.410719 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.400653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46252 (local address 127.0.98.60:32835)
0619 12:32:31.400775 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:31.400779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:31.400792 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:32:31.400909 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:31.400912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:31.400959 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:31.401030 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:31.401466 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.402378 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.403718 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.404030 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.404169 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.406223 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:31.406226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:31.406229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:31.406251 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:31.408769 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.409307 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.409310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.409312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.409423 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.409750 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.409754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.409755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.409988 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:31.410152 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:31.410450 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 12:32:31.410567 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44,"spinlock_wait_cycles":896}
I20260619 12:32:31.413353 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.403569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32835 (local address 127.0.0.1:46258)
0619 12:32:31.403686 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.403711 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:31.403781 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:31.404314 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:31.404317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:31.404485 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:31.404558 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:31.404563 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.406117 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.406123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:31.406785 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:31.406793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.407272 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.407275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:31.407386 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.407823 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:31.407838 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:31.409144 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:31.411764 (+ 2620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:31.411767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:31.411769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:31.412003 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:31.412342 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:31.412345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:31.412347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:31.412419 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:31.412806 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:31.412809 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:31.412893 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 12:32:31.413002 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 12:32:31.413181 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":21}
I20260619 12:32:31.413666 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.403660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46258 (local address 127.0.98.60:32835)
0619 12:32:31.404036 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:31.404046 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:31.404060 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:32:31.404088 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:31.404091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:31.404131 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:31.404227 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:31.405264 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.406019 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.406910 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.407163 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.407265 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.409277 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:31.409283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:31.409288 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:31.409318 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:31.411633 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.412165 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.412168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.412170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.412248 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.412509 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.412514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.412515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.412706 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:31.412816 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:31.413075 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 12:32:31.413506 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":313,"spinlock_wait_cycles":7424,"thread_start_us":152,"threads_started":1}
I20260619 12:32:31.414013 13784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc: Bootstrap starting.
I20260619 12:32:31.414480 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.404755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52368 (local address 127.0.98.61:45009)
0619 12:32:31.404864 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:31.404868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:31.404881 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:32:31.404914 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:31.404917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:31.404978 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:31.405035 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:31.405799 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.406516 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.407263 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.407639 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.407752 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.410610 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:31.410613 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:31.410616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:31.410640 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:31.412438 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.412890 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.412893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.412895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.412979 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.413388 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.413393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.413395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.413609 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:31.413793 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:31.414191 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 12:32:31.414303 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 12:32:31.414994 13859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } attempt: 3
I20260619 12:32:31.416491 13934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead: Bootstrap starting.
I20260619 12:32:31.417651 13784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc: Neither blocks nor log segments found. Creating new log.
I20260619 12:32:31.417871 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:32:31.418627 13784 log.cc:824] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:31.419389 13934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead: Neither blocks nor log segments found. Creating new log.
I20260619 12:32:31.420329 13934 log.cc:824] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead: Log is configured to *not* fsync() on all Append() calls
Jun 19 12:32:31 dist-test-slave-fg3h krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 12:32:31.422771 13934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead: No bootstrap required, opened a new log
I20260619 12:32:31.422772 13784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc: No bootstrap required, opened a new log
I20260619 12:32:31.424332 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.415328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32835 (local address 127.0.0.1:46264)
0619 12:32:31.415448 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:31.415462 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:31.415526 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:31.415957 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:31.415960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:31.416130 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:31.416236 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:31.416243 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.417268 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.417273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:31.418044 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:31.418053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.418710 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.418714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:31.418796 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.419293 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:31.419311 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:31.420849 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:31.422809 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:31.422812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:31.422814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:31.423056 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:31.423345 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:31.423348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:31.423350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:31.423433 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:31.423858 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:31.423861 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:31.423945 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 12:32:31.424046 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 12:32:31.424175 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 12:32:31.424903 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.415640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46264 (local address 127.0.98.60:32835)
0619 12:32:31.415747 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:31.415751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:31.415762 (+ 11us) server_negotiation.cc:407] Connection header received
0619 12:32:31.415790 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:31.415794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:31.415835 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:31.416338 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:31.416417 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.417147 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.418186 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.418568 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.419000 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.420974 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:31.420978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:31.420980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:31.421003 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:31.422691 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.423165 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.423168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.423170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.423249 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.423529 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.423532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.423534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.423737 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:31.423817 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:31.424611 (+ 794us) server_negotiation.cc:299] Negotiation successful
0619 12:32:31.424725 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:32:31.427301 13784 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } } peers { permanent_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } }
I20260619 12:32:31.427518 13784 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:32:31.427565 13784 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfeee5a6372e4767a6ef4e3ac04ff1bc, State: Initialized, Role: FOLLOWER
I20260619 12:32:31.427785 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80: Bootstrap starting.
I20260619 12:32:31.428036 13784 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } } peers { permanent_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } }
I20260619 12:32:31.428536 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } } peers { permanent_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } } }
I20260619 12:32:31.428711 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:32:31.429428 13784 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:32:31.429675 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.420154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.0.98.62:42863)
0619 12:32:31.420535 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:31.420539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:31.420804 (+ 265us) server_negotiation.cc:407] Connection header received
0619 12:32:31.420986 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:31.420990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:31.421035 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:31.421122 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:31.422168 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.422931 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.424066 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.424384 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.424465 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.426101 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:31.426105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:31.426107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:31.426131 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:31.427650 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.428059 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.428064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.428066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.428150 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.428466 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.428469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.428471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.428680 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:31.428800 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:31.429264 (+ 464us) server_negotiation.cc:299] Negotiation successful
0619 12:32:31.429384 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308}
I20260619 12:32:31.431253 13859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80: Neither blocks nor log segments found. Creating new log.
I20260619 12:32:31.432287 13859 log.cc:824] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:31.434676 13934 raft_consensus.cc:348] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } } peers { permanent_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } }
I20260619 12:32:31.434938 13934 raft_consensus.cc:374] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:32:31.434993 13934 raft_consensus.cc:729] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad8dbfb8c8c413eb51cf86b80d72ead, State: Initialized, Role: FOLLOWER
I20260619 12:32:31.435469 13934 consensus_queue.cc:260] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } } peers { permanent_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } }
I20260619 12:32:31.436859 13934 sys_catalog.cc:564] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:32:31.438966 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.420234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46278 (local address 127.0.98.60:32835)
0619 12:32:31.420337 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:31.420341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:31.429650 (+ 9309us) server_negotiation.cc:407] Connection header received
0619 12:32:31.429848 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:31.429851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:31.429892 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:31.429987 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:31.431192 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.431982 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.432903 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.433214 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.433312 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.435040 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:31.435045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:31.435047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:31.435075 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:31.436792 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.437312 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.437318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.437320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.437401 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.437722 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.437729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.437733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.437972 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:31.438122 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:31.438705 (+ 583us) server_negotiation.cc:299] Negotiation successful
0619 12:32:31.438815 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 12:32:31.435279 13716 master_runner.cc:186] Error getting master registration for 127.0.98.61:45009: OK, instance_id { permanent_uuid: "e1842df1675546c888091f5042a91a80" instance_seqno: 1781872351155554 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 12:32:31.440394 13716 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 12:32:31.440691 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80: No bootstrap required, opened a new log
I20260619 12:32:31.442405 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:31.420584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52380 (local address 127.0.98.61:45009)
0619 12:32:31.420685 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:31.420689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:31.433443 (+ 12754us) server_negotiation.cc:407] Connection header received
0619 12:32:31.433679 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:31.433682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:31.433726 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:31.433779 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:31.434677 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.435395 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.436604 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:31.436907 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:31.436990 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:31.438828 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:31.438833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:31.438836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:31.438867 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:31.440530 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.441162 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.441165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.441167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.441257 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:31.441533 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:31.441536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:31.441538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:31.441746 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:31.441871 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:31.442157 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 12:32:31.442267 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 12:32:31.444132 13960 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:32:31.444227 13960 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:32:31.446588 13859 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } } peers { permanent_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } }
I20260619 12:32:31.446823 13859 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:32:31.446885 13859 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1842df1675546c888091f5042a91a80, State: Initialized, Role: FOLLOWER
I20260619 12:32:31.447343 13859 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } } peers { permanent_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } }
I20260619 12:32:31.447809 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } } peers { permanent_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } } }
I20260619 12:32:31.447995 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:32:31.448634 13859 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 12:32:31.452006 13974 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:32:31.452091 13974 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:32:31.452337 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } } peers { permanent_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } } }
I20260619 12:32:31.452445 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:32:31.448949 13865 master_runner.cc:186] Error getting master registration for 127.0.98.61:45009: OK, instance_id { permanent_uuid: "e1842df1675546c888091f5042a91a80" instance_seqno: 1781872351155554 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 12:32:31.453688 13865 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 12:32:31.456161 13791 master_runner.cc:263] No leader master found from master e1842df1675546c888091f5042a91a80
I20260619 12:32:31.456239 13791 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 12:32:31.460692 13987 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:32:31.460814 13987 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:32:31.594480 13962 raft_consensus.cc:482] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:32:31.594682 13962 raft_consensus.cc:504] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } } peers { permanent_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } }
I20260619 12:32:31.595772 13962 leader_election.cc:288] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfeee5a6372e4767a6ef4e3ac04ff1bc (127.0.98.62:42863), e1842df1675546c888091f5042a91a80 (127.0.98.61:45009)
I20260619 12:32:31.596557 13758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" is_pre_election: true
I20260619 12:32:31.596882 13758 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad8dbfb8c8c413eb51cf86b80d72ead in term 0.
I20260619 12:32:31.597589 13833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1842df1675546c888091f5042a91a80" is_pre_election: true
I20260619 12:32:31.597899 13833 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad8dbfb8c8c413eb51cf86b80d72ead in term 0.
I20260619 12:32:31.598321 13885 leader_election.cc:302] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad8dbfb8c8c413eb51cf86b80d72ead, e1842df1675546c888091f5042a91a80; no voters:
I20260619 12:32:31.598877 13962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:32:31.598981 13962 raft_consensus.cc:482] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:32:31.599043 13962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:32:31.600358 13962 raft_consensus.cc:504] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } } peers { permanent_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } }
I20260619 12:32:31.600729 13962 leader_election.cc:288] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [CANDIDATE]: Term 1 election: Requested vote from peers dfeee5a6372e4767a6ef4e3ac04ff1bc (127.0.98.62:42863), e1842df1675546c888091f5042a91a80 (127.0.98.61:45009)
I20260619 12:32:31.601213 13758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc"
I20260619 12:32:31.601380 13758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:32:31.602739 13758 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad8dbfb8c8c413eb51cf86b80d72ead in term 1.
I20260619 12:32:31.603111 13885 leader_election.cc:302] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad8dbfb8c8c413eb51cf86b80d72ead, dfeee5a6372e4767a6ef4e3ac04ff1bc; no voters:
I20260619 12:32:31.603430 13833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1842df1675546c888091f5042a91a80"
I20260619 12:32:31.603554 13833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:32:31.606315 13833 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad8dbfb8c8c413eb51cf86b80d72ead in term 1.
I20260619 12:32:31.606635 13962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:32:31.606863 13962 raft_consensus.cc:686] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [term 1 LEADER]: Becoming Leader. State: Replica: aad8dbfb8c8c413eb51cf86b80d72ead, State: Running, Role: LEADER
I20260619 12:32:31.607228 13962 consensus_queue.cc:237] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } } peers { permanent_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } }
I20260619 12:32:31.609215 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [sys.catalog]: SysCatalogTable state changed. Reason: New leader aad8dbfb8c8c413eb51cf86b80d72ead. Latest consensus state: current_term: 1 leader_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } } peers { permanent_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } } }
I20260619 12:32:31.609330 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [sys.catalog]: This master's current role is: LEADER
I20260619 12:32:31.613310 13991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:32:31.614049 13991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 12:32:31.616145 13887 tablet.cc:2369] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:32:31.620108 13833 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80 [term 1 FOLLOWER]: Refusing update from remote peer aad8dbfb8c8c413eb51cf86b80d72ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:32:31.620108 13758 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc [term 1 FOLLOWER]: Refusing update from remote peer aad8dbfb8c8c413eb51cf86b80d72ead: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:32:31.620769 13989 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:32:31.620918 13962 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:32:31.625135 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aad8dbfb8c8c413eb51cf86b80d72ead. Latest consensus state: current_term: 1 leader_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } } peers { permanent_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } } }
I20260619 12:32:31.625149 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader aad8dbfb8c8c413eb51cf86b80d72ead. Latest consensus state: current_term: 1 leader_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } } peers { permanent_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } } }
I20260619 12:32:31.625285 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:32:31.625293 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:32:31.631610 13989 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } } peers { permanent_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } } }
I20260619 12:32:31.631757 13989 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [sys.catalog]: This master's current role is: LEADER
I20260619 12:32:31.631875 13989 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } } peers { permanent_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } } }
I20260619 12:32:31.631961 13989 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead [sys.catalog]: This master's current role is: LEADER
I20260619 12:32:31.632043 13992 mvcc.cc:205] Tried to move back new op lower bound from 7298549152228257792 to 7298549152187129856. Current Snapshot: MvccSnapshot[applied={T|T < 7298549152228257792}]
I20260619 12:32:31.632278 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } } peers { permanent_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } } }
I20260619 12:32:31.632392 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:32:31.632526 13994 mvcc.cc:205] Tried to move back new op lower bound from 7298549152228257792 to 7298549152187129856. Current Snapshot: MvccSnapshot[applied={T|T < 7298549152228257792}]
I20260619 12:32:31.633327 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfeee5a6372e4767a6ef4e3ac04ff1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "e1842df1675546c888091f5042a91a80" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45009 } } peers { permanent_uuid: "aad8dbfb8c8c413eb51cf86b80d72ead" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32835 } } }
I20260619 12:32:31.633435 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:32:31.634989 13991 catalog_manager.cc:1347] Generated new cluster ID: cfd1c6a38ec24b41ab7ac62d4b6b99c3
I20260619 12:32:31.635053 13991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:32:31.648703 13991 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:32:31.650173 13991 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:32:31.658178 13991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aad8dbfb8c8c413eb51cf86b80d72ead: Generated new TSK 0
I20260619 12:32:31.659006 13991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:32:32.445569 13960 catalog_manager.cc:1259] Loaded cluster ID: cfd1c6a38ec24b41ab7ac62d4b6b99c3
I20260619 12:32:32.445662 13960 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc: loading cluster ID for follower catalog manager: success
I20260619 12:32:32.447363 13960 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc: acquiring CA information for follower catalog manager: success
I20260619 12:32:32.448813 13960 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dfeee5a6372e4767a6ef4e3ac04ff1bc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:32:32.462208 13987 catalog_manager.cc:1259] Loaded cluster ID: cfd1c6a38ec24b41ab7ac62d4b6b99c3
I20260619 12:32:32.462327 13987 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80: loading cluster ID for follower catalog manager: success
I20260619 12:32:32.464116 13987 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80: acquiring CA information for follower catalog manager: success
I20260619 12:32:32.465472 13987 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e1842df1675546c888091f5042a91a80: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:32:32.720649 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 13716
I20260619 12:32:32.727351 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 13791
I20260619 12:32:32.733162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 13865
2026-06-19T12:32:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3159 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:32:32 dist-test-slave-fg3h krb5kdc[14006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:32:32 dist-test-slave-fg3h krb5kdc[14006](info): set up 2 sockets
Jun 19 12:32:32 dist-test-slave-fg3h krb5kdc[14006](info): commencing operation
krb5kdc: starting...
W20260619 12:32:33.793479 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:32:33 dist-test-slave-fg3h krb5kdc[14006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:32:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:32:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:32:33.915592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42925
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:32:34.024266 14019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:34.024569 14019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:34.024652 14019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:34.028640 14019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:32:34.028752 14019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:34.028786 14019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:34.028825 14019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:32:34.028857 14019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:32:34.034260 14019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:34.035557 14019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:34.036691 14019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:34.045202 14027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:34.045295 14024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:34.045305 14025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:34.045471 14019 server_base.cc:1034] running on GCE node
I20260619 12:32:34.046087 14019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:34.047205 14019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:34.048404 14019 hybrid_clock.cc:648] HybridClock initialized: now 1781872354048391 us; error 28 us; skew 500 ppm
Jun 19 12:32:34 dist-test-slave-fg3h krb5kdc[14006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:34.051323 14019 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:32:34.052158 14019 webserver.cc:460] Webserver started at http://127.0.98.62:41337/ using document root <none> and password file <none>
I20260619 12:32:34.052687 14019 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:34.052747 14019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:34.052908 14019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:32:34.055054 14019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "7bc52154bfc542c19e34682680af28be"
format_stamp: "Formatted at 2026-06-19 12:32:34 on dist-test-slave-fg3h"
server_key: "d77a8a2211a3cad38d22727d8bbe2305"
server_key_iv: "75af939175fa3b6ff7d72914436a8936"
server_key_version: "encryptionkey@0"
I20260619 12:32:34.055631 14019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "7bc52154bfc542c19e34682680af28be"
format_stamp: "Formatted at 2026-06-19 12:32:34 on dist-test-slave-fg3h"
server_key: "d77a8a2211a3cad38d22727d8bbe2305"
server_key_iv: "75af939175fa3b6ff7d72914436a8936"
server_key_version: "encryptionkey@0"
I20260619 12:32:34.059372 14019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:32:34.061867 14034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:34.063105 14019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:32:34.063205 14019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "7bc52154bfc542c19e34682680af28be"
format_stamp: "Formatted at 2026-06-19 12:32:34 on dist-test-slave-fg3h"
server_key: "d77a8a2211a3cad38d22727d8bbe2305"
server_key_iv: "75af939175fa3b6ff7d72914436a8936"
server_key_version: "encryptionkey@0"
I20260619 12:32:34.063344 14019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:34.087090 14019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:34.090548 14019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:34.090773 14019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:34.098212 14019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42925
I20260619 12:32:34.098240 14086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42925 every 8 connection(s)
I20260619 12:32:34.099231 14019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:32:34.101949 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 14019
I20260619 12:32:34.102077 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:32:34.102186 14087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:32:34.102418 392 external_mini_cluster.cc:1428] Setting key fd50a0083b89e0f9a7085857a194092f
I20260619 12:32:34.109576 14087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be: Bootstrap starting.
I20260619 12:32:34.111948 14087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be: Neither blocks nor log segments found. Creating new log.
Jun 19 12:32:34 dist-test-slave-fg3h krb5kdc[14006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:32:34.112725 14087 log.cc:824] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:34.115167 14087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be: No bootstrap required, opened a new log
I20260619 12:32:34.118858 14090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:34.103923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52700 (local address 127.0.98.62:42925)
0619 12:32:34.104377 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:34.104399 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:34.104444 (+ 45us) server_negotiation.cc:407] Connection header received
0619 12:32:34.105661 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:34.105687 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:34.106135 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:34.106523 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:34.107244 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:34.108723 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:34.109769 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:34.110278 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:34.110473 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:34.113370 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:34.113375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:34.113388 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:34.113414 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:34.115526 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:34.115996 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:34.116002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:34.116008 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:34.116159 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:34.116464 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:34.116468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:34.116470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:34.116978 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:34.117224 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:34.117526 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 12:32:34.117880 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":140,"threads_started":1}
I20260619 12:32:34.119843 14087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc52154bfc542c19e34682680af28be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42925 } }
I20260619 12:32:34.120131 14087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:32:34.120193 14087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc52154bfc542c19e34682680af28be, State: Initialized, Role: FOLLOWER
I20260619 12:32:34.120685 14087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc52154bfc542c19e34682680af28be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42925 } }
I20260619 12:32:34.120858 14087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:32:34.120954 14087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:32:34.121055 14087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:32:34.121958 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:32:34.122192 14087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc52154bfc542c19e34682680af28be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42925 } }
I20260619 12:32:34.122529 14087 leader_election.cc:302] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bc52154bfc542c19e34682680af28be; no voters:
I20260619 12:32:34.122890 14087 leader_election.cc:288] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:32:34.122964 14092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:32:34.123193 14092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be [term 1 LEADER]: Becoming Leader. State: Replica: 7bc52154bfc542c19e34682680af28be, State: Running, Role: LEADER
I20260619 12:32:34.123543 14092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc52154bfc542c19e34682680af28be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42925 } }
I20260619 12:32:34.124239 14087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:32:34.125377 14093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bc52154bfc542c19e34682680af28be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc52154bfc542c19e34682680af28be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42925 } } }
I20260619 12:32:34.125506 14093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be [sys.catalog]: This master's current role is: LEADER
I20260619 12:32:34.125823 14094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bc52154bfc542c19e34682680af28be. Latest consensus state: current_term: 1 leader_uuid: "7bc52154bfc542c19e34682680af28be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bc52154bfc542c19e34682680af28be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42925 } } }
I20260619 12:32:34.125947 14101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:32:34.125975 14094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be [sys.catalog]: This master's current role is: LEADER
I20260619 12:32:34.129194 14101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:32:34.135092 14101 catalog_manager.cc:1347] Generated new cluster ID: a016363a832848e7bd77abcece5238e0
I20260619 12:32:34.135169 14101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:32:34.153908 14101 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:32:34.155596 14101 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:32:34.160126 14101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bc52154bfc542c19e34682680af28be: Generated new TSK 0
I20260619 12:32:34.160854 14101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:32:34.177857 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 14019
2026-06-19T12:32:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1442 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:32:34 dist-test-slave-fg3h krb5kdc[14114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:32:34 dist-test-slave-fg3h krb5kdc[14114](info): set up 2 sockets
Jun 19 12:32:34 dist-test-slave-fg3h krb5kdc[14114](info): commencing operation
krb5kdc: starting...
W20260619 12:32:35.277642 392 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:32:35 dist-test-slave-fg3h krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:32:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:32:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:32:35.395977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45791
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:32:35.505046 14127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:35.505375 14127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:35.505458 14127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:35.509507 14127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:32:35.509585 14127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:35.509621 14127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:35.509682 14127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:32:35.509720 14127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:32:35.515014 14127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:35.516065 14127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:35.517122 14127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:35.525491 14135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:35.525561 14127 server_base.cc:1034] running on GCE node
W20260619 12:32:35.525501 14133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:35.525501 14132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:35.526386 14127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:35.527422 14127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:35.528622 14127 hybrid_clock.cc:648] HybridClock initialized: now 1781872355528613 us; error 30 us; skew 500 ppm
Jun 19 12:32:35 dist-test-slave-fg3h krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:35.536854 14127 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:32:35.537851 14127 webserver.cc:460] Webserver started at http://127.0.98.62:45085/ using document root <none> and password file <none>
I20260619 12:32:35.538431 14127 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:35.538522 14127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:35.538753 14127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:32:35.540793 14127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "215ae7f9577a4b548d66e18c413d43f7"
format_stamp: "Formatted at 2026-06-19 12:32:35 on dist-test-slave-fg3h"
server_key: "cdd7055645a1c0f7cf95fc10d6ae0a05"
server_key_iv: "e3dd887944ea267ab1a9e647ad54b5e6"
server_key_version: "encryptionkey@0"
I20260619 12:32:35.541384 14127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "215ae7f9577a4b548d66e18c413d43f7"
format_stamp: "Formatted at 2026-06-19 12:32:35 on dist-test-slave-fg3h"
server_key: "cdd7055645a1c0f7cf95fc10d6ae0a05"
server_key_iv: "e3dd887944ea267ab1a9e647ad54b5e6"
server_key_version: "encryptionkey@0"
I20260619 12:32:35.545019 14127 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 12:32:35.547482 14142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:35.548776 14127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 12:32:35.548926 14127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "215ae7f9577a4b548d66e18c413d43f7"
format_stamp: "Formatted at 2026-06-19 12:32:35 on dist-test-slave-fg3h"
server_key: "cdd7055645a1c0f7cf95fc10d6ae0a05"
server_key_iv: "e3dd887944ea267ab1a9e647ad54b5e6"
server_key_version: "encryptionkey@0"
I20260619 12:32:35.549043 14127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:35.558924 14127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:35.563527 14127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:35.563768 14127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:35.571591 14127 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45791
I20260619 12:32:35.571862 14194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45791 every 8 connection(s)
I20260619 12:32:35.572678 14127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:32:35.575505 14195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:32:35.580991 14195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7: Bootstrap starting.
I20260619 12:32:35.581969 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 14127
I20260619 12:32:35.582137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:32:35.582491 392 external_mini_cluster.cc:1428] Setting key e7fd2f7c6f8beadde5bfd63afc84202f
I20260619 12:32:35.583879 14195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7: Neither blocks nor log segments found. Creating new log.
I20260619 12:32:35.584642 14195 log.cc:824] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:35.586894 14195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7: No bootstrap required, opened a new log
I20260619 12:32:35.590349 14195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215ae7f9577a4b548d66e18c413d43f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260619 12:32:35.590642 14195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:32:35.590721 14195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 215ae7f9577a4b548d66e18c413d43f7, State: Initialized, Role: FOLLOWER
I20260619 12:32:35.591264 14195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215ae7f9577a4b548d66e18c413d43f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260619 12:32:35.591439 14195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:32:35.591521 14195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:32:35.591635 14195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 12:32:35 dist-test-slave-fg3h krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:32:35.592725 14195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215ae7f9577a4b548d66e18c413d43f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260619 12:32:35.593118 14195 leader_election.cc:302] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 215ae7f9577a4b548d66e18c413d43f7; no voters:
I20260619 12:32:35.593487 14195 leader_election.cc:288] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:32:35.593562 14200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:32:35.593770 14200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 1 LEADER]: Becoming Leader. State: Replica: 215ae7f9577a4b548d66e18c413d43f7, State: Running, Role: LEADER
I20260619 12:32:35.594110 14200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215ae7f9577a4b548d66e18c413d43f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260619 12:32:35.594857 14195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:32:35.597772 14201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "215ae7f9577a4b548d66e18c413d43f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215ae7f9577a4b548d66e18c413d43f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } } }
I20260619 12:32:35.597908 14201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [sys.catalog]: This master's current role is: LEADER
I20260619 12:32:35.598292 14215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:32:35.599756 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:35.583718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.0.98.62:45791)
0619 12:32:35.584154 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:35.584170 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:35.584206 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:32:35.584964 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:35.584983 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:35.585405 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:35.585706 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:35.586747 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:35.587885 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:35.589150 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:35.590194 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:35.590329 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:35.593307 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:35.593314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:35.593331 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:35.593369 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:35.595723 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:35.596267 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:35.596272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:35.596278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:35.596416 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:35.596773 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:35.596778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:35.596780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:35.597319 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:35.597530 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:35.598474 (+ 944us) server_negotiation.cc:299] Negotiation successful
0619 12:32:35.598798 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":143,"threads_started":1}
I20260619 12:32:35.601169 14202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 215ae7f9577a4b548d66e18c413d43f7. Latest consensus state: current_term: 1 leader_uuid: "215ae7f9577a4b548d66e18c413d43f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215ae7f9577a4b548d66e18c413d43f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } } }
I20260619 12:32:35.601279 14202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [sys.catalog]: This master's current role is: LEADER
I20260619 12:32:35.601622 14215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:32:35.607080 14215 catalog_manager.cc:1347] Generated new cluster ID: d0c48f9743a04229a1aa67e468b02c6f
I20260619 12:32:35.607141 14215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:32:35.629827 14215 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:32:35.630941 14215 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:32:35.637459 14215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7: Generated new TSK 0
I20260619 12:32:35.638250 14215 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:32:35.667959 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45791
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:32:35.779353 14221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:35.779657 14221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:35.779750 14221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:35.783815 14221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:35.783901 14221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:35.783989 14221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:32:35.789131 14221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:35.790174 14221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:35.791172 14221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:35.801433 14229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:35.801313 14226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:35.801313 14227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:35.801616 14221 server_base.cc:1034] running on GCE node
I20260619 12:32:35.802091 14221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:35.802712 14221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:35.803897 14221 hybrid_clock.cc:648] HybridClock initialized: now 1781872355803876 us; error 33 us; skew 500 ppm
Jun 19 12:32:35 dist-test-slave-fg3h krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:35.806648 14221 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:32:35.807539 14221 webserver.cc:460] Webserver started at http://127.0.98.1:34381/ using document root <none> and password file <none>
I20260619 12:32:35.808094 14221 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:35.808200 14221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:35.808429 14221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:32:35.810462 14221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "72a029cb92664b639ed82b023679810a"
format_stamp: "Formatted at 2026-06-19 12:32:35 on dist-test-slave-fg3h"
server_key: "fb0a68cfadb919ef0ccf419a2efa575f"
server_key_iv: "65edf6272a95d181179cd15575505ec1"
server_key_version: "encryptionkey@0"
I20260619 12:32:35.811014 14221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "72a029cb92664b639ed82b023679810a"
format_stamp: "Formatted at 2026-06-19 12:32:35 on dist-test-slave-fg3h"
server_key: "fb0a68cfadb919ef0ccf419a2efa575f"
server_key_iv: "65edf6272a95d181179cd15575505ec1"
server_key_version: "encryptionkey@0"
I20260619 12:32:35.814648 14221 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:32:35.816934 14236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:35.818317 14221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:32:35.818452 14221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "72a029cb92664b639ed82b023679810a"
format_stamp: "Formatted at 2026-06-19 12:32:35 on dist-test-slave-fg3h"
server_key: "fb0a68cfadb919ef0ccf419a2efa575f"
server_key_iv: "65edf6272a95d181179cd15575505ec1"
server_key_version: "encryptionkey@0"
I20260619 12:32:35.818563 14221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:35.847309 14221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:35.851001 14221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:35.851235 14221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:35.851881 14221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:32:35.852895 14221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:32:35.852972 14221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:35.853050 14221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:32:35.853134 14221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 12:32:35.862478 14221 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40125
I20260619 12:32:35.862536 14349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40125 every 8 connection(s)
I20260619 12:32:35.863431 14221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:32:35.865052 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 14221
I20260619 12:32:35.865239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:32:35.865595 392 external_mini_cluster.cc:1428] Setting key d12042e5879333c526e56bb004d07d75
Jun 19 12:32:35 dist-test-slave-fg3h krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:32:35.879156 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:35.865542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59619 (local address 127.0.98.62:45791)
0619 12:32:35.865721 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:35.865726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:35.866664 (+ 938us) server_negotiation.cc:407] Connection header received
0619 12:32:35.868471 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:35.868473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:35.868517 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:35.868581 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:35.869998 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:35.870543 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:35.871805 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:35.872054 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:35.872173 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:35.874960 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:35.874964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:35.874967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:35.874997 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:35.876554 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:35.877236 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:35.877240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:35.877242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:35.877357 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:35.877837 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:35.877840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:35.877843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:35.878120 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:35.878238 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:35.878797 (+ 559us) server_negotiation.cc:299] Negotiation successful
0619 12:32:35.878949 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 12:32:35.880268 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:35.865916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:59619)
0619 12:32:35.866485 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:35.866544 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:35.868236 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:35.868794 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:35.868806 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:35.869337 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:35.869842 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:35.869857 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:35.870681 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:35.870684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:35.871665 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:35.871675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:35.872206 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:35.872210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:35.872396 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:35.873027 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:35.873043 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:35.874737 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:35.876735 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:35.876741 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:35.876753 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:35.877041 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:35.877499 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:35.877506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:35.877510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:35.877716 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:35.878263 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:35.878268 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:35.878533 (+ 265us) client_negotiation.cc:769] Sending connection context
0619 12:32:35.878695 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 12:32:35.879058 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":133,"threads_started":1}
I20260619 12:32:35.881698 14350 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45791
I20260619 12:32:35.881969 14350 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:35.882477 14350 heartbeater.cc:511] Master 127.0.98.62:45791 requested a full tablet report, sending...
I20260619 12:32:35.884060 14159 ts_manager.cc:194] Registered new tserver with Master: 72a029cb92664b639ed82b023679810a (127.0.98.1:40125)
I20260619 12:32:35.885421 14159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:32:35.894239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45791
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:32:36.006031 14355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:36.006316 14355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:36.006399 14355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:36.011015 14355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:36.011118 14355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:36.011237 14355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:32:36.017047 14355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:36.018144 14355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:36.019150 14355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:36.028744 14363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:36.028784 14361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:36.028744 14360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:36.029004 14355 server_base.cc:1034] running on GCE node
I20260619 12:32:36.029709 14355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:36.030407 14355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:36.031642 14355 hybrid_clock.cc:648] HybridClock initialized: now 1781872356031613 us; error 49 us; skew 500 ppm
Jun 19 12:32:36 dist-test-slave-fg3h krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:36.034723 14355 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:32:36.035729 14355 webserver.cc:460] Webserver started at http://127.0.98.2:35629/ using document root <none> and password file <none>
I20260619 12:32:36.036276 14355 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:36.036337 14355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:36.036510 14355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:32:36.038697 14355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "a574e055cc414b6fbadfdb5c7290082f"
format_stamp: "Formatted at 2026-06-19 12:32:36 on dist-test-slave-fg3h"
server_key: "7e7ca55982dbe12113a829b812879218"
server_key_iv: "b223b659fd99ef04953b38a2951b97f8"
server_key_version: "encryptionkey@0"
I20260619 12:32:36.039248 14355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a574e055cc414b6fbadfdb5c7290082f"
format_stamp: "Formatted at 2026-06-19 12:32:36 on dist-test-slave-fg3h"
server_key: "7e7ca55982dbe12113a829b812879218"
server_key_iv: "b223b659fd99ef04953b38a2951b97f8"
server_key_version: "encryptionkey@0"
I20260619 12:32:36.043205 14355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:32:36.045787 14370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:36.047205 14355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 12:32:36.047335 14355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "a574e055cc414b6fbadfdb5c7290082f"
format_stamp: "Formatted at 2026-06-19 12:32:36 on dist-test-slave-fg3h"
server_key: "7e7ca55982dbe12113a829b812879218"
server_key_iv: "b223b659fd99ef04953b38a2951b97f8"
server_key_version: "encryptionkey@0"
I20260619 12:32:36.047451 14355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:36.061177 14355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:36.064438 14355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:36.064644 14355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:36.065316 14355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:32:36.066218 14355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:32:36.066275 14355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:36.066326 14355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:32:36.066361 14355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:36.075848 14355 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40917
I20260619 12:32:36.075867 14483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40917 every 8 connection(s)
I20260619 12:32:36.076869 14355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:32:36.080925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 14355
I20260619 12:32:36.081053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:32:36.081395 392 external_mini_cluster.cc:1428] Setting key 54568f73a8f1cb0b3982039238adb832
Jun 19 12:32:36 dist-test-slave-fg3h krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:32:36.092319 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:36.078956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54595 (local address 127.0.98.62:45791)
0619 12:32:36.079102 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:36.079105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:36.079915 (+ 810us) server_negotiation.cc:407] Connection header received
0619 12:32:36.081440 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:36.081444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:36.081528 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:36.081612 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:36.083093 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.083713 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.085334 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.085660 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.085789 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:36.088514 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:36.088518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:36.088520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:36.088549 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:36.090210 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:36.090735 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:36.090742 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:36.090745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:36.090827 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:36.091168 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:36.091173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:36.091174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:36.091404 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:36.091510 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:36.092022 (+ 512us) server_negotiation.cc:299] Negotiation successful
0619 12:32:36.092150 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:32:36.093353 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:36.079251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:54595)
0619 12:32:36.079746 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:36.079789 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:36.081181 (+ 1392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:36.081814 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:36.081822 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:36.082296 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:36.082891 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:36.082909 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.083875 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.083882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:36.085206 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:36.085217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.085844 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.085848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:36.086024 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:36.086722 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:36.086738 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:36.088337 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:36.090334 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:36.090341 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:36.090357 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:36.090628 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:36.090926 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:36.090929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:36.090931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:36.091072 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:36.091515 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:36.091524 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:36.091772 (+ 248us) client_negotiation.cc:769] Sending connection context
0619 12:32:36.091961 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 12:32:36.092326 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 12:32:36.094606 14484 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45791
I20260619 12:32:36.094918 14484 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:36.095472 14484 heartbeater.cc:511] Master 127.0.98.62:45791 requested a full tablet report, sending...
I20260619 12:32:36.096627 14159 ts_manager.cc:194] Registered new tserver with Master: a574e055cc414b6fbadfdb5c7290082f (127.0.98.2:40917)
I20260619 12:32:36.097304 14159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:32:36.110203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45791
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:32:36.219497 14489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:36.219771 14489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:36.219815 14489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:36.223860 14489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:36.223923 14489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:36.223982 14489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:32:36.229455 14489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:36.230427 14489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:36.231382 14489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:36.241070 14497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:36.241111 14494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:36.241117 14495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:36.241410 14489 server_base.cc:1034] running on GCE node
I20260619 12:32:36.241794 14489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:36.242354 14489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:36.243510 14489 hybrid_clock.cc:648] HybridClock initialized: now 1781872356243496 us; error 25 us; skew 500 ppm
Jun 19 12:32:36 dist-test-slave-fg3h krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:36.246253 14489 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:32:36.247138 14489 webserver.cc:460] Webserver started at http://127.0.98.3:40367/ using document root <none> and password file <none>
I20260619 12:32:36.247691 14489 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:36.247781 14489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:36.248005 14489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:32:36.250087 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133"
format_stamp: "Formatted at 2026-06-19 12:32:36 on dist-test-slave-fg3h"
server_key: "f146a8ca929b7b10a2c866ade53d0eac"
server_key_iv: "73f50742f64b870c5fb95dfb6788353d"
server_key_version: "encryptionkey@0"
I20260619 12:32:36.250656 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133"
format_stamp: "Formatted at 2026-06-19 12:32:36 on dist-test-slave-fg3h"
server_key: "f146a8ca929b7b10a2c866ade53d0eac"
server_key_iv: "73f50742f64b870c5fb95dfb6788353d"
server_key_version: "encryptionkey@0"
I20260619 12:32:36.254365 14489 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:32:36.256754 14504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:36.258036 14489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:32:36.258165 14489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133"
format_stamp: "Formatted at 2026-06-19 12:32:36 on dist-test-slave-fg3h"
server_key: "f146a8ca929b7b10a2c866ade53d0eac"
server_key_iv: "73f50742f64b870c5fb95dfb6788353d"
server_key_version: "encryptionkey@0"
I20260619 12:32:36.258275 14489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:36.284650 14489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:36.288249 14489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:36.288489 14489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:36.289142 14489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:32:36.290000 14489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:32:36.290071 14489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:36.290145 14489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:32:36.290189 14489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:36.299834 14489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33871
I20260619 12:32:36.299854 14617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33871 every 8 connection(s)
I20260619 12:32:36.300765 14489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:32:36.306703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 14489
I20260619 12:32:36.306854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:32:36.307127 392 external_mini_cluster.cc:1428] Setting key db6c82e0b8b1513a88e24c87cf172486
Jun 19 12:32:36 dist-test-slave-fg3h krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:32:36.314975 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:36.302636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34519 (local address 127.0.98.62:45791)
0619 12:32:36.302813 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:36.302819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:36.303852 (+ 1033us) server_negotiation.cc:407] Connection header received
0619 12:32:36.305151 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:36.305155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:36.305219 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:36.305301 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:36.306490 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.307127 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.308633 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.308859 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.308973 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:36.311601 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:36.311605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:36.311610 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:36.311640 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:36.312865 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:36.313441 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:36.313445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:36.313447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:36.313531 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:36.313879 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:36.313884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:36.313886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:36.314067 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:36.314194 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:36.314689 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 12:32:36.314808 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 12:32:36.315778 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:36.303095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:34519)
0619 12:32:36.303669 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:36.303730 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:36.304862 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:36.305483 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:36.305490 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:36.305852 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:36.306347 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:36.306360 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.307266 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.307272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:36.308507 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:36.308517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.309012 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.309017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:36.309242 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:36.309957 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:36.309973 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:36.311430 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:36.312985 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:36.312991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:36.313006 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:36.313335 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:36.313631 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:36.313634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:36.313644 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:36.313782 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:36.314193 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:36.314202 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:36.314446 (+ 244us) client_negotiation.cc:769] Sending connection context
0619 12:32:36.314643 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 12:32:36.314959 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":119,"threads_started":1}
I20260619 12:32:36.316903 14618 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45791
I20260619 12:32:36.317180 14618 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:36.317622 14618 heartbeater.cc:511] Master 127.0.98.62:45791 requested a full tablet report, sending...
I20260619 12:32:36.318612 14159 ts_manager.cc:194] Registered new tserver with Master: 7671644a7cbd4d3cb9fc7f6b44c7e133 (127.0.98.3:33871)
I20260619 12:32:36.319172 14159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34519
I20260619 12:32:36.321102 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:32:36.330590 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:36.323301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40188 (local address 127.0.98.62:45791)
0619 12:32:36.323438 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:36.323442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:36.323594 (+ 152us) server_negotiation.cc:407] Connection header received
0619 12:32:36.323680 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:36.323683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:36.323719 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:36.323780 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:36.324398 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.324901 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.325849 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.326058 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.326155 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:36.327504 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:36.327507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:36.327509 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:36.327530 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:36.329032 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:36.329554 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:36.329556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:36.329558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:36.329633 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:36.329877 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:36.329879 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:36.329880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:36.330087 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:36.330210 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:36.330378 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 12:32:36.330469 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 12:32:36.333527 14159 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:32:36.336562 14159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:32:36.350741 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:36.344569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40125 (local address 127.0.0.1:37454)
0619 12:32:36.344911 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:36.344960 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:36.345078 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:36.346520 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:36.346522 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:36.346536 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:32:36.346643 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:36.346649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.348003 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.348006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:36.349166 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:36.349173 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.350104 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.350109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:36.350249 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:36.350306 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 12:32:36.350409 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 12:32:36.350538 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":36,"thread_start_us":75,"threads_started":1}
I20260619 12:32:36.351070 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:36.345232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37454 (local address 127.0.98.1:40125)
0619 12:32:36.346099 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:36.346111 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:36.346130 (+ 19us) server_negotiation.cc:407] Connection header received
0619 12:32:36.346215 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:36.346220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:36.346399 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:36.346558 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:32:36.346733 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.347867 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.349273 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.349982 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.350070 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:36.350177 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:36.350834 (+ 657us) server_negotiation.cc:299] Negotiation successful
0619 12:32:36.350941 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":68,"threads_started":1}
I20260619 12:32:36.351260 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:36.344941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33871 (local address 127.0.0.1:38026)
0619 12:32:36.346117 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:36.346130 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:36.346229 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:36.347021 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:36.347024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:36.347043 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:32:36.347126 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:36.347132 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.348423 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.348427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:36.349523 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:36.349534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.350368 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.350372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:36.350519 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:36.350545 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 12:32:36.350653 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 12:32:36.350725 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1119,"mutex_wait_us":25,"thread_start_us":102,"threads_started":1}
I20260619 12:32:36.351269 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:36.345583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38026 (local address 127.0.98.3:33871)
0619 12:32:36.345943 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:36.345956 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:36.346227 (+ 271us) server_negotiation.cc:407] Connection header received
0619 12:32:36.346695 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:36.346703 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:36.346881 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:36.347023 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:32:36.347226 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.348288 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.349650 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.350241 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.350349 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:36.350495 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:36.350674 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 12:32:36.350775 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":186,"threads_started":1}
I20260619 12:32:36.352349 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:36.344768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40917 (local address 127.0.0.1:44484)
0619 12:32:36.345464 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:36.345481 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:36.345571 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:36.346334 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:36.346340 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:36.346362 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:32:36.346519 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:36.346526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.347704 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.347710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:36.349152 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:36.349163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.351230 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.351234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:36.352084 (+ 850us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:36.352112 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 12:32:36.352182 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 12:32:36.352252 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":50,"thread_start_us":102,"threads_started":1}
I20260619 12:32:36.352919 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:36.345342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44484 (local address 127.0.98.2:40917)
0619 12:32:36.345812 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:36.345825 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:36.345854 (+ 29us) server_negotiation.cc:407] Connection header received
0619 12:32:36.345938 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:36.345943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:36.346194 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:36.346350 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:32:36.346654 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.347577 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.350227 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.351104 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.352532 (+ 1428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:36.352678 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:36.352730 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 12:32:36.352814 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":119,"threads_started":1}
I20260619 12:32:36.354252 14552 tablet_service.cc:1467] Processing CreateTablet for tablet 5bed466cb0bb4801bdbacbe334c9693d (DEFAULT_TABLE table=test-table [id=50e2111f478a4d35aaae3246a234896f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:32:36.354252 14284 tablet_service.cc:1467] Processing CreateTablet for tablet 5bed466cb0bb4801bdbacbe334c9693d (DEFAULT_TABLE table=test-table [id=50e2111f478a4d35aaae3246a234896f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:32:36.355391 14552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bed466cb0bb4801bdbacbe334c9693d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:32:36.355387 14284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bed466cb0bb4801bdbacbe334c9693d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:32:36.355525 14418 tablet_service.cc:1467] Processing CreateTablet for tablet 5bed466cb0bb4801bdbacbe334c9693d (DEFAULT_TABLE table=test-table [id=50e2111f478a4d35aaae3246a234896f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:32:36.356458 14418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bed466cb0bb4801bdbacbe334c9693d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:32:36.362457 14635 tablet_bootstrap.cc:492] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f: Bootstrap starting.
I20260619 12:32:36.363684 14636 tablet_bootstrap.cc:492] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a: Bootstrap starting.
I20260619 12:32:36.363684 14637 tablet_bootstrap.cc:492] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133: Bootstrap starting.
I20260619 12:32:36.365010 14635 tablet_bootstrap.cc:654] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f: Neither blocks nor log segments found. Creating new log.
I20260619 12:32:36.365871 14637 tablet_bootstrap.cc:654] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133: Neither blocks nor log segments found. Creating new log.
I20260619 12:32:36.365871 14636 tablet_bootstrap.cc:654] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a: Neither blocks nor log segments found. Creating new log.
I20260619 12:32:36.365918 14635 log.cc:824] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:36.367172 14636 log.cc:824] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:36.367168 14637 log.cc:824] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:36.367947 14635 tablet_bootstrap.cc:492] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f: No bootstrap required, opened a new log
I20260619 12:32:36.368151 14635 ts_tablet_manager.cc:1397] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:32:36.369153 14637 tablet_bootstrap.cc:492] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133: No bootstrap required, opened a new log
I20260619 12:32:36.369339 14637 ts_tablet_manager.cc:1397] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:32:36.369453 14636 tablet_bootstrap.cc:492] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a: No bootstrap required, opened a new log
I20260619 12:32:36.369632 14636 ts_tablet_manager.cc:1397] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:32:36.371356 14635 raft_consensus.cc:348] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } }
I20260619 12:32:36.371615 14635 raft_consensus.cc:374] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:32:36.371697 14635 raft_consensus.cc:729] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a574e055cc414b6fbadfdb5c7290082f, State: Initialized, Role: FOLLOWER
I20260619 12:32:36.372148 14635 consensus_queue.cc:260] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } }
I20260619 12:32:36.373201 14635 ts_tablet_manager.cc:1428] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 12:32:36.373338 14636 raft_consensus.cc:348] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } }
I20260619 12:32:36.373338 14637 raft_consensus.cc:348] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } }
I20260619 12:32:36.373564 14484 heartbeater.cc:503] Master 127.0.98.62:45791 was elected leader, sending a full tablet report...
I20260619 12:32:36.373595 14636 raft_consensus.cc:374] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:32:36.373595 14637 raft_consensus.cc:374] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:32:36.373656 14636 raft_consensus.cc:729] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a029cb92664b639ed82b023679810a, State: Initialized, Role: FOLLOWER
I20260619 12:32:36.373656 14637 raft_consensus.cc:729] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7671644a7cbd4d3cb9fc7f6b44c7e133, State: Initialized, Role: FOLLOWER
I20260619 12:32:36.374089 14637 consensus_queue.cc:260] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } }
I20260619 12:32:36.374089 14636 consensus_queue.cc:260] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } }
I20260619 12:32:36.374751 14350 heartbeater.cc:503] Master 127.0.98.62:45791 was elected leader, sending a full tablet report...
I20260619 12:32:36.375039 14618 heartbeater.cc:503] Master 127.0.98.62:45791 was elected leader, sending a full tablet report...
I20260619 12:32:36.375057 14637 ts_tablet_manager.cc:1428] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:32:36.375995 14636 ts_tablet_manager.cc:1428] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260619 12:32:36.551834 14619 tablet.cc:2369] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:32:36.578168 14485 tablet.cc:2369] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:32:36.614905 14351 tablet.cc:2369] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:32:36.669210 14641 raft_consensus.cc:482] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:32:36.669423 14641 raft_consensus.cc:504] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } }
I20260619 12:32:36.670588 14641 leader_election.cc:288] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72a029cb92664b639ed82b023679810a (127.0.98.1:40125), 7671644a7cbd4d3cb9fc7f6b44c7e133 (127.0.98.3:33871)
I20260619 12:32:36.675911 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:36.670923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47675 (local address 127.0.98.1:40125)
0619 12:32:36.671162 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:36.671169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:36.671233 (+ 64us) server_negotiation.cc:407] Connection header received
0619 12:32:36.671398 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:36.671403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:36.671456 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:36.671551 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:32:36.671916 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.672708 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.674551 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.675185 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.675299 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:36.675364 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:36.675681 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 12:32:36.675768 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260619 12:32:36.675911 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:36.671008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41627 (local address 127.0.98.3:33871)
0619 12:32:36.671143 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:36.671148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:36.671351 (+ 203us) server_negotiation.cc:407] Connection header received
0619 12:32:36.671424 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:36.671427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:36.671475 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:36.671558 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:32:36.671916 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.672733 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.674585 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.675188 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.675299 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:36.675365 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:36.675681 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 12:32:36.675768 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 12:32:36.675956 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:36.670829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40125 (local address 127.0.98.2:47675)
0619 12:32:36.671115 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:36.671148 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:36.671274 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:36.671585 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:36.671588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:36.671629 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:32:36.671793 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:36.671800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.672842 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.672845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:36.674409 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:36.674421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.675317 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.675321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:36.675490 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:36.675533 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 12:32:36.675614 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 12:32:36.675765 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":289,"spinlock_wait_cycles":29696,"thread_start_us":89,"threads_started":1}
I20260619 12:32:36.675958 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:36.670917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33871 (local address 127.0.98.2:41627)
0619 12:32:36.671242 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:36.671254 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:36.671347 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:36.671585 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:36.671588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:36.671656 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:32:36.671797 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:36.671802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.672842 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.672845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:36.674413 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:36.674421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.675317 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.675321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:36.675494 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:36.675535 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 12:32:36.675614 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 12:32:36.675766 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":213,"spinlock_wait_cycles":6784,"thread_start_us":65,"threads_started":1}
I20260619 12:32:36.676939 14304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bed466cb0bb4801bdbacbe334c9693d" candidate_uuid: "a574e055cc414b6fbadfdb5c7290082f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72a029cb92664b639ed82b023679810a" is_pre_election: true
I20260619 12:32:36.676936 14572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bed466cb0bb4801bdbacbe334c9693d" candidate_uuid: "a574e055cc414b6fbadfdb5c7290082f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" is_pre_election: true
I20260619 12:32:36.677289 14304 raft_consensus.cc:2454] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a574e055cc414b6fbadfdb5c7290082f in term 0.
I20260619 12:32:36.677294 14572 raft_consensus.cc:2454] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a574e055cc414b6fbadfdb5c7290082f in term 0.
I20260619 12:32:36.677798 14372 leader_election.cc:302] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 72a029cb92664b639ed82b023679810a, 7671644a7cbd4d3cb9fc7f6b44c7e133, a574e055cc414b6fbadfdb5c7290082f; no voters:
I20260619 12:32:36.678095 14641 raft_consensus.cc:2781] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:32:36.678188 14641 raft_consensus.cc:482] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:32:36.678224 14641 raft_consensus.cc:3037] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:32:36.679358 14641 raft_consensus.cc:504] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } }
I20260619 12:32:36.679793 14641 leader_election.cc:288] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [CANDIDATE]: Term 1 election: Requested vote from peers 72a029cb92664b639ed82b023679810a (127.0.98.1:40125), 7671644a7cbd4d3cb9fc7f6b44c7e133 (127.0.98.3:33871)
I20260619 12:32:36.680182 14304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bed466cb0bb4801bdbacbe334c9693d" candidate_uuid: "a574e055cc414b6fbadfdb5c7290082f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72a029cb92664b639ed82b023679810a"
I20260619 12:32:36.680258 14572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bed466cb0bb4801bdbacbe334c9693d" candidate_uuid: "a574e055cc414b6fbadfdb5c7290082f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133"
I20260619 12:32:36.680326 14304 raft_consensus.cc:3037] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:32:36.680399 14572 raft_consensus.cc:3037] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:32:36.681460 14304 raft_consensus.cc:2454] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a574e055cc414b6fbadfdb5c7290082f in term 1.
I20260619 12:32:36.681603 14572 raft_consensus.cc:2454] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a574e055cc414b6fbadfdb5c7290082f in term 1.
I20260619 12:32:36.681844 14371 leader_election.cc:302] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72a029cb92664b639ed82b023679810a, a574e055cc414b6fbadfdb5c7290082f; no voters:
I20260619 12:32:36.682030 14641 raft_consensus.cc:2781] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:32:36.682271 14641 raft_consensus.cc:686] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [term 1 LEADER]: Becoming Leader. State: Replica: a574e055cc414b6fbadfdb5c7290082f, State: Running, Role: LEADER
I20260619 12:32:36.682613 14641 consensus_queue.cc:237] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } }
I20260619 12:32:36.685842 14157 catalog_manager.cc:5495] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f reported cstate change: term changed from 0 to 1, leader changed from <none> to a574e055cc414b6fbadfdb5c7290082f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a574e055cc414b6fbadfdb5c7290082f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } health_report { overall_health: UNKNOWN } } }
I20260619 12:32:36.712276 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:36.708728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44488 (local address 127.0.98.2:40917)
0619 12:32:36.708859 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:36.708863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:36.708876 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:32:36.708958 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:36.708960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:36.709016 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:36.709177 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:32:36.709474 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.710014 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.710857 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:36.711081 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:36.711176 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:36.711484 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:32:36.711900 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:32:36.711988 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:36.712114 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 12:32:36.712168 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:32:36.718266 14304 raft_consensus.cc:1261] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Refusing update from remote peer a574e055cc414b6fbadfdb5c7290082f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:32:36.718266 14572 raft_consensus.cc:1261] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 1 FOLLOWER]: Refusing update from remote peer a574e055cc414b6fbadfdb5c7290082f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:32:36.719012 14641 consensus_queue.cc:1035] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:32:36.719210 14646 consensus_queue.cc:1035] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [LEADER]: Connected to new peer: Peer: permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:32:36.735540 14158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:32:36.735718 14158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:32:36.738003 14158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7: Sending DeleteTablet for 3 replicas of tablet 5bed466cb0bb4801bdbacbe334c9693d
I20260619 12:32:36.738852 14552 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bed466cb0bb4801bdbacbe334c9693d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:32:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38026
I20260619 12:32:36.738933 14284 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bed466cb0bb4801bdbacbe334c9693d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:32:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37454
I20260619 12:32:36.739068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 14221
I20260619 12:32:36.739367 14418 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bed466cb0bb4801bdbacbe334c9693d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:32:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44484
I20260619 12:32:36.740368 14658 tablet_replica.cc:330] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133: stopping tablet replica
I20260619 12:32:36.740710 14658 raft_consensus.cc:2229] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:32:36.741052 14658 raft_consensus.cc:2258] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:32:36.741268 14659 tablet_replica.cc:330] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f: stopping tablet replica
I20260619 12:32:36.742012 14659 raft_consensus.cc:2229] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:32:36.742395 14659 raft_consensus.cc:2258] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:32:36.742502 14658 ts_tablet_manager.cc:1905] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:32:36.743726 14659 ts_tablet_manager.cc:1905] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:32:36.745251 14658 ts_tablet_manager.cc:1918] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:32:36.745352 14658 log.cc:1196] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/5bed466cb0bb4801bdbacbe334c9693d
I20260619 12:32:36.745712 14658 ts_tablet_manager.cc:1939] T 5bed466cb0bb4801bdbacbe334c9693d P 7671644a7cbd4d3cb9fc7f6b44c7e133: Deleting consensus metadata
I20260619 12:32:36.746488 14659 ts_tablet_manager.cc:1918] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:32:36.746585 14659 log.cc:1196] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/5bed466cb0bb4801bdbacbe334c9693d
I20260619 12:32:36.746906 14659 ts_tablet_manager.cc:1939] T 5bed466cb0bb4801bdbacbe334c9693d P a574e055cc414b6fbadfdb5c7290082f: Deleting consensus metadata
I20260619 12:32:36.747036 14144 catalog_manager.cc:4841] TS 7671644a7cbd4d3cb9fc7f6b44c7e133 (127.0.98.3:33871): tablet 5bed466cb0bb4801bdbacbe334c9693d (table test-table [id=50e2111f478a4d35aaae3246a234896f]) successfully deleted
W20260619 12:32:36.747546 14143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40125 (error 108)
I20260619 12:32:36.748031 14145 catalog_manager.cc:4841] TS a574e055cc414b6fbadfdb5c7290082f (127.0.98.2:40917): tablet 5bed466cb0bb4801bdbacbe334c9693d (table test-table [id=50e2111f478a4d35aaae3246a234896f]) successfully deleted
I20260619 12:32:36.748572 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 14355
I20260619 12:32:36.748667 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:36.748368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40125 (local address 127.0.0.1:37468)
0619 12:32:36.748494 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:36.748569 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 12:32:36.748816 14143 catalog_manager.cc:4584] TS 72a029cb92664b639ed82b023679810a (127.0.98.1:40125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5bed466cb0bb4801bdbacbe334c9693d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40125: connect: Connection refused (error 111)
I20260619 12:32:36.754694 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 14489
I20260619 12:32:36.760269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 14127
I20260619 12:32:36.766537 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45791
--webserver_interface=127.0.98.62
--webserver_port=45085
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:32:36.880215 14661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:36.880518 14661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:36.880560 14661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:36.884881 14661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:32:36.884956 14661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:36.884976 14661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:36.884994 14661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:32:36.885008 14661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:32:36.890599 14661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:36.891693 14661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:36.892760 14661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:36.905985 14668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:36.906050 14670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:36.905997 14667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:36.906241 14661 server_base.cc:1034] running on GCE node
I20260619 12:32:36.906884 14661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:36.908141 14661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:36.909372 14661 hybrid_clock.cc:648] HybridClock initialized: now 1781872356909350 us; error 46 us; skew 500 ppm
Jun 19 12:32:36 dist-test-slave-fg3h krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:36.912365 14661 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:32:36.913362 14661 webserver.cc:460] Webserver started at http://127.0.98.62:45085/ using document root <none> and password file <none>
I20260619 12:32:36.913913 14661 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:36.913985 14661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:36.917950 14661 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:32:36.920253 14677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:36.921623 14661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:32:36.921732 14661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "215ae7f9577a4b548d66e18c413d43f7"
format_stamp: "Formatted at 2026-06-19 12:32:35 on dist-test-slave-fg3h"
server_key: "cdd7055645a1c0f7cf95fc10d6ae0a05"
server_key_iv: "e3dd887944ea267ab1a9e647ad54b5e6"
server_key_version: "encryptionkey@0"
I20260619 12:32:36.922415 14661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:36.944689 14661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:36.948796 14661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:36.948962 14661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:36.956439 14661 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45791
I20260619 12:32:36.956758 14729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45791 every 8 connection(s)
I20260619 12:32:36.957666 14661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:32:36.962414 14730 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:32:36.963152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 14661
I20260619 12:32:36.963644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40125
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45791
--webserver_port=34381
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:32:36.964373 14730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7: Bootstrap starting.
I20260619 12:32:36.987660 14730 log.cc:824] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:36.996451 14730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:32:36.996871 14730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7: Bootstrap complete.
I20260619 12:32:37.001204 14730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215ae7f9577a4b548d66e18c413d43f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260619 12:32:37.001642 14730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 215ae7f9577a4b548d66e18c413d43f7, State: Initialized, Role: FOLLOWER
I20260619 12:32:37.002185 14730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215ae7f9577a4b548d66e18c413d43f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260619 12:32:37.002331 14730 raft_consensus.cc:388] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:32:37.002424 14730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:32:37.002525 14730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:32:37.004438 14730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215ae7f9577a4b548d66e18c413d43f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260619 12:32:37.004786 14730 leader_election.cc:302] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 215ae7f9577a4b548d66e18c413d43f7; no voters:
I20260619 12:32:37.005168 14730 leader_election.cc:288] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 12:32:37.005264 14734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 12:32:37.005494 14734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 2 LEADER]: Becoming Leader. State: Replica: 215ae7f9577a4b548d66e18c413d43f7, State: Running, Role: LEADER
I20260619 12:32:37.005777 14734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215ae7f9577a4b548d66e18c413d43f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260619 12:32:37.006403 14730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:32:37.006713 14736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 215ae7f9577a4b548d66e18c413d43f7. Latest consensus state: current_term: 2 leader_uuid: "215ae7f9577a4b548d66e18c413d43f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215ae7f9577a4b548d66e18c413d43f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } } }
I20260619 12:32:37.006843 14736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [sys.catalog]: This master's current role is: LEADER
I20260619 12:32:37.007066 14735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "215ae7f9577a4b548d66e18c413d43f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215ae7f9577a4b548d66e18c413d43f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } } }
I20260619 12:32:37.007170 14735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [sys.catalog]: This master's current role is: LEADER
I20260619 12:32:37.007488 14738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:32:37.012353 14738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:32:37.013230 14738 catalog_manager.cc:1259] Loaded cluster ID: d0c48f9743a04229a1aa67e468b02c6f
I20260619 12:32:37.013293 14738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:32:37.014755 14738 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 12:32:37.019654 14738 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 12:32:37.025295 14738 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:32:37.026479 14738 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7: Loaded TSK: 0
I20260619 12:32:37.027704 14738 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 12:32:37.089864 14732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:37.090180 14732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:37.090269 14732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:37.094547 14732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:37.094700 14732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:37.094801 14732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:32:37.100756 14732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34381
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:37.101959 14732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:37.103034 14732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:37.113022 14760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:37.113099 14758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:37.113049 14757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:37.114893 14732 server_base.cc:1034] running on GCE node
I20260619 12:32:37.115417 14732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:37.116246 14732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:37.117491 14732 hybrid_clock.cc:648] HybridClock initialized: now 1781872357117363 us; error 91 us; skew 500 ppm
Jun 19 12:32:37 dist-test-slave-fg3h krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:37.121065 14732 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:32:37.122323 14732 webserver.cc:460] Webserver started at http://127.0.98.1:34381/ using document root <none> and password file <none>
I20260619 12:32:37.123054 14732 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:37.123167 14732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:37.128655 14732 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 12:32:37.131541 14767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:37.133200 14732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 12:32:37.133329 14732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "72a029cb92664b639ed82b023679810a"
format_stamp: "Formatted at 2026-06-19 12:32:35 on dist-test-slave-fg3h"
server_key: "fb0a68cfadb919ef0ccf419a2efa575f"
server_key_iv: "65edf6272a95d181179cd15575505ec1"
server_key_version: "encryptionkey@0"
I20260619 12:32:37.134063 14732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:37.153457 14732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:37.157481 14732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:37.157698 14732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:37.158409 14732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:32:37.159880 14774 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 12:32:37.164203 14732 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 12:32:37.164287 14732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260619 12:32:37.164351 14732 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 12:32:37.166589 14732 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 12:32:37.166651 14732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 12:32:37.167062 14774 tablet_bootstrap.cc:492] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a: Bootstrap starting.
I20260619 12:32:37.177205 14732 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40125
I20260619 12:32:37.177297 14881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40125 every 8 connection(s)
I20260619 12:32:37.178232 14732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:32:37.181341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 14732
I20260619 12:32:37.181830 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40917
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45791
--webserver_port=35629
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:32:37.189978 14774 log.cc:824] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a: Log is configured to *not* fsync() on all Append() calls
Jun 19 12:32:37 dist-test-slave-fg3h krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:32:37.199781 14774 tablet_bootstrap.cc:492] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:32:37.200299 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:37.180528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:52985)
0619 12:32:37.181195 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:37.181257 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:37.182955 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:37.184292 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:37.184300 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:37.184759 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:37.185339 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:37.185356 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.187398 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.187402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:37.188943 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:37.188957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.190258 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.190263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:37.190499 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:37.191281 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:37.191309 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:37.193534 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:37.196460 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:37.196467 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:37.196480 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:37.196832 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:37.197347 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:37.197351 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:37.197353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:37.197523 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:37.198351 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:37.198358 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:37.198665 (+ 307us) client_negotiation.cc:769] Sending connection context
0619 12:32:37.198852 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 12:32:37.199193 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":177,"threads_started":1}
I20260619 12:32:37.200295 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:37.180430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52985 (local address 127.0.98.62:45791)
0619 12:32:37.180827 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:37.180843 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:37.181526 (+ 683us) server_negotiation.cc:407] Connection header received
0619 12:32:37.183564 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:37.183576 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:37.183889 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:37.184240 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:37.185531 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.187180 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.189139 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.190035 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.190243 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:37.193798 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:37.193805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:37.193823 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:37.193861 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:37.196296 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:37.197000 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:37.197008 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:37.197014 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:37.197230 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:37.197669 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:37.197673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:37.197675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:37.198227 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:37.198364 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:37.199048 (+ 684us) server_negotiation.cc:299] Negotiation successful
0619 12:32:37.199380 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":158,"threads_started":1}
I20260619 12:32:37.200989 14774 tablet_bootstrap.cc:492] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a: Bootstrap complete.
I20260619 12:32:37.202330 14774 ts_tablet_manager.cc:1397] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a: Time spent bootstrapping tablet: real 0.036s user 0.029s sys 0.004s
I20260619 12:32:37.203500 14882 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45791
I20260619 12:32:37.203745 14882 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:37.204325 14882 heartbeater.cc:511] Master 127.0.98.62:45791 requested a full tablet report, sending...
I20260619 12:32:37.205338 14774 raft_consensus.cc:348] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } }
I20260619 12:32:37.206012 14774 raft_consensus.cc:729] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a029cb92664b639ed82b023679810a, State: Initialized, Role: FOLLOWER
I20260619 12:32:37.206512 14774 consensus_queue.cc:260] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } }
I20260619 12:32:37.207343 14694 ts_manager.cc:194] Registered new tserver with Master: 72a029cb92664b639ed82b023679810a (127.0.98.1:40125)
I20260619 12:32:37.207712 14774 ts_tablet_manager.cc:1428] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 12:32:37.211421 14694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52985
I20260619 12:32:37.214159 14882 heartbeater.cc:503] Master 127.0.98.62:45791 was elected leader, sending a full tablet report...
Jun 19 12:32:37 dist-test-slave-fg3h krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:32:37.222545 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:37.210356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40125 (local address 127.0.0.1:37478)
0619 12:32:37.211160 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:37.211208 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:37.211401 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:37.212106 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:37.212111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:37.212551 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:37.212682 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:37.212691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.214051 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.214054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:37.214972 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:37.214985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.215910 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.215926 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:37.216010 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:37.216642 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:37.216656 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:37.218394 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:37.220676 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:37.220685 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:37.220689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:37.220923 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:37.221299 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:37.221302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:37.221304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:37.221399 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:37.221936 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:37.221941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:37.222049 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 12:32:37.222189 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 12:32:37.222350 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":109,"threads_started":1}
I20260619 12:32:37.222846 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:37.210444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.0.98.1:40125)
0619 12:32:37.210776 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:37.210788 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:37.211295 (+ 507us) server_negotiation.cc:407] Connection header received
0619 12:32:37.211582 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:37.211588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:37.211780 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:37.211920 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:37.212843 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.213915 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.215117 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.215624 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.215738 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:37.218557 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:37.218566 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:37.218579 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:37.218607 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:37.220536 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:37.221049 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:37.221057 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:37.221061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:37.221191 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:37.221501 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:37.221505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:37.221507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:37.221812 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:37.221978 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:37.222293 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 12:32:37.222467 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":138,"threads_started":1}
I20260619 12:32:37.223760 14816 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bed466cb0bb4801bdbacbe334c9693d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:32:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37478
I20260619 12:32:37.224640 14891 tablet_replica.cc:330] stopping tablet replica
I20260619 12:32:37.224954 14891 raft_consensus.cc:2229] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:32:37.225256 14891 raft_consensus.cc:2258] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:32:37.226406 14891 ts_tablet_manager.cc:1905] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:32:37.231073 14891 ts_tablet_manager.cc:1918] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:32:37.231189 14891 log.cc:1196] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/wal/wals/5bed466cb0bb4801bdbacbe334c9693d
I20260619 12:32:37.231459 14891 ts_tablet_manager.cc:1939] T 5bed466cb0bb4801bdbacbe334c9693d P 72a029cb92664b639ed82b023679810a: Deleting consensus metadata
I20260619 12:32:37.233577 14678 catalog_manager.cc:4841] TS 72a029cb92664b639ed82b023679810a (127.0.98.1:40125): tablet 5bed466cb0bb4801bdbacbe334c9693d (table test-table [id=50e2111f478a4d35aaae3246a234896f]) successfully deleted
W20260619 12:32:37.308195 14886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:37.308565 14886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:37.308645 14886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:37.312796 14886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:37.312853 14886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:37.312950 14886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:32:37.318305 14886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35629
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:37.319350 14886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:37.320372 14886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:37.330056 14896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:37.330027 14899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:37.330049 14897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:37.330240 14886 server_base.cc:1034] running on GCE node
I20260619 12:32:37.330889 14886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:37.331506 14886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:37.332681 14886 hybrid_clock.cc:648] HybridClock initialized: now 1781872357332649 us; error 48 us; skew 500 ppm
Jun 19 12:32:37 dist-test-slave-fg3h krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:37.335444 14886 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:32:37.336349 14886 webserver.cc:460] Webserver started at http://127.0.98.2:35629/ using document root <none> and password file <none>
I20260619 12:32:37.336925 14886 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:37.337033 14886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:37.341004 14886 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:32:37.343098 14906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:37.344300 14886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:32:37.344434 14886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "a574e055cc414b6fbadfdb5c7290082f"
format_stamp: "Formatted at 2026-06-19 12:32:36 on dist-test-slave-fg3h"
server_key: "7e7ca55982dbe12113a829b812879218"
server_key_iv: "b223b659fd99ef04953b38a2951b97f8"
server_key_version: "encryptionkey@0"
I20260619 12:32:37.344995 14886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:37.368881 14886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:37.372454 14886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:37.372689 14886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:37.373337 14886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:32:37.374176 14886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:32:37.374245 14886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:37.374320 14886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:32:37.374362 14886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:37.383700 14886 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40917
I20260619 12:32:37.383723 15019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40917 every 8 connection(s)
I20260619 12:32:37.384728 14886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:32:37.390692 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 14886
I20260619 12:32:37.391149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33871
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45791
--webserver_port=40367
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:32:37 dist-test-slave-fg3h krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:32:37.400223 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:37.386764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39433 (local address 127.0.98.62:45791)
0619 12:32:37.386943 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:37.386948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:37.387755 (+ 807us) server_negotiation.cc:407] Connection header received
0619 12:32:37.389013 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:37.389016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:37.389065 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:37.389171 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:37.390486 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.391346 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.392597 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.392838 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.392963 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:37.395622 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:37.395628 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:37.395633 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:37.395674 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:37.397645 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:37.398357 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:37.398364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:37.398369 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:37.398477 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:37.398837 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:37.398841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:37.398846 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:37.399143 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:37.399263 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:37.399805 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 12:32:37.399965 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 12:32:37.400977 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:37.387100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:39433)
0619 12:32:37.387593 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:37.387638 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:37.388783 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:37.389361 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:37.389368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:37.389804 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:37.390303 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:37.390317 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.391503 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.391507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:37.392447 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:37.392458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.393005 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.393009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:37.393268 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:37.393934 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:37.393951 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:37.395420 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:37.397772 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:37.397777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:37.397792 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:37.398139 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:37.398580 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:37.398582 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:37.398584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:37.398734 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:37.399260 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:37.399266 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:37.399519 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 12:32:37.399716 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 12:32:37.399995 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":111,"threads_started":1}
I20260619 12:32:37.402330 15020 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45791
I20260619 12:32:37.402576 15020 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:37.403098 15020 heartbeater.cc:511] Master 127.0.98.62:45791 requested a full tablet report, sending...
I20260619 12:32:37.404202 14694 ts_manager.cc:194] Registered new tserver with Master: a574e055cc414b6fbadfdb5c7290082f (127.0.98.2:40917)
I20260619 12:32:37.404798 14694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39433
W20260619 12:32:37.505972 15023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:37.506246 15023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:37.506325 15023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:37.510910 15023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:37.510974 15023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:37.511066 15023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:32:37.516556 15023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40367
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:37.517722 15023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:37.518751 15023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:37.528918 15029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:37.528918 15028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:37.528983 15031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:37.529373 15023 server_base.cc:1034] running on GCE node
I20260619 12:32:37.529922 15023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:37.530609 15023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:37.531826 15023 hybrid_clock.cc:648] HybridClock initialized: now 1781872357531787 us; error 50 us; skew 500 ppm
Jun 19 12:32:37 dist-test-slave-fg3h krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:37.534965 15023 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:32:37.535914 15023 webserver.cc:460] Webserver started at http://127.0.98.3:40367/ using document root <none> and password file <none>
I20260619 12:32:37.536520 15023 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:37.536635 15023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:37.540656 15023 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:32:37.543077 15038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:37.544389 15023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 12:32:37.544590 15023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133"
format_stamp: "Formatted at 2026-06-19 12:32:36 on dist-test-slave-fg3h"
server_key: "f146a8ca929b7b10a2c866ade53d0eac"
server_key_iv: "73f50742f64b870c5fb95dfb6788353d"
server_key_version: "encryptionkey@0"
I20260619 12:32:37.545207 15023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:37.562463 15023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:37.567194 15023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:37.567420 15023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:37.568048 15023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:32:37.568890 15023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:32:37.568944 15023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:37.569047 15023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:32:37.569116 15023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:37.578610 15023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33871
I20260619 12:32:37.578637 15151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33871 every 8 connection(s)
I20260619 12:32:37.579612 15023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:32:37.588023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 15023
Jun 19 12:32:37 dist-test-slave-fg3h krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:32:37.594370 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:37.581483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33719 (local address 127.0.98.62:45791)
0619 12:32:37.581633 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:37.581637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:37.582496 (+ 859us) server_negotiation.cc:407] Connection header received
0619 12:32:37.583834 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:37.583837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:37.583885 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:37.583963 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:37.585269 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.585816 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.587080 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.587301 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.587392 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:37.590468 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:37.590472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:37.590475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:37.590508 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:37.592101 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:37.592676 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:37.592679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:37.592681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:37.592769 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:37.593189 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:37.593194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:37.593196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:37.593383 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:37.593499 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:37.594059 (+ 560us) server_negotiation.cc:299] Negotiation successful
0619 12:32:37.594181 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 12:32:37.595245 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:37.581894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:33719)
0619 12:32:37.582334 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:37.582379 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:37.583612 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:37.584155 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:37.584162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:37.584547 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:37.585054 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:37.585068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.585949 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.585952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:37.586942 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:37.586951 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.587471 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.587474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:37.587657 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:37.588341 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:37.588359 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:37.590280 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:37.592230 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:37.592237 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:37.592255 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:37.592552 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:37.592876 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:37.592879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:37.592881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:37.593042 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:37.593502 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:37.593512 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:37.593777 (+ 265us) client_negotiation.cc:769] Sending connection context
0619 12:32:37.593977 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 12:32:37.594332 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":105,"threads_started":1}
I20260619 12:32:37.596661 15152 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45791
I20260619 12:32:37.596906 15152 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:37.597615 15152 heartbeater.cc:511] Master 127.0.98.62:45791 requested a full tablet report, sending...
I20260619 12:32:37.598032 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:37.588788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40190 (local address 127.0.98.62:45791)
0619 12:32:37.589070 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:37.589074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:37.589215 (+ 141us) server_negotiation.cc:407] Connection header received
0619 12:32:37.589371 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:37.589375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:37.589477 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:37.589583 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:37.590382 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.591057 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.592054 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.592293 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.592410 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:37.593769 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:37.593772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:37.593775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:37.593798 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:37.595756 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:37.596539 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:37.596541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:37.596543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:37.596674 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:37.597107 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:37.597111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:37.597112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:37.597364 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:37.597586 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:37.597739 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 12:32:37.597860 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":122,"threads_started":1}
I20260619 12:32:37.598822 14694 ts_manager.cc:194] Registered new tserver with Master: 7671644a7cbd4d3cb9fc7f6b44c7e133 (127.0.98.3:33871)
I20260619 12:32:37.599630 14694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33719
I20260619 12:32:37.599988 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:32:37.604319 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:37.600624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40202 (local address 127.0.98.62:45791)
0619 12:32:37.600729 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:37.600733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:37.600972 (+ 239us) server_negotiation.cc:407] Connection header received
0619 12:32:37.601108 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:37.601111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:37.601219 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:37.601324 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:32:37.601582 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.602195 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.603081 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.603296 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.603378 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:37.603696 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:32:37.604035 (+ 339us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:32:37.604122 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:37.604181 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 12:32:37.604227 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 12:32:37.604923 14694 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:32:37.608493 14694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:32:37.617753 14816 tablet_service.cc:1467] Processing CreateTablet for tablet 00a03174fd07450f8307b41e01a0f109 (DEFAULT_TABLE table=test-table [id=cd0d682a125d4a3e941e09649afc60a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:32:37.618470 14816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a03174fd07450f8307b41e01a0f109. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:32:37.621762 14774 tablet_bootstrap.cc:492] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a: Bootstrap starting.
I20260619 12:32:37.622133 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:37.616076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33871 (local address 127.0.0.1:38032)
0619 12:32:37.616185 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:37.616204 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:37.616284 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:37.617710 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:37.617713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:37.617730 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:32:37.617855 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:37.617861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.619237 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.619241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:37.620574 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:37.620585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.621662 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.621685 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:37.621787 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:37.621831 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 12:32:37.621889 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 12:32:37.621992 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 12:32:37.622372 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:37.616444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40917 (local address 127.0.0.1:44504)
0619 12:32:37.616709 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:37.616724 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:37.616809 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:37.617923 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:37.617929 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:37.617948 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:32:37.618058 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:37.618065 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.619643 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.619649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:37.620439 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:37.620449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.621989 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.621992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:37.622091 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:37.622111 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 12:32:37.622176 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 12:32:37.622246 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":128,"threads_started":1}
I20260619 12:32:37.622732 14774 tablet_bootstrap.cc:654] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a: Neither blocks nor log segments found. Creating new log.
I20260619 12:32:37.622776 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:37.616503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38032 (local address 127.0.98.3:33871)
0619 12:32:37.617254 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:37.617267 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:37.617289 (+ 22us) server_negotiation.cc:407] Connection header received
0619 12:32:37.617365 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:37.617369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:37.617551 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:37.617723 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:32:37.617976 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.619120 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.620701 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.621518 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.622328 (+ 810us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:37.622492 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:37.622549 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 12:32:37.622644 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":160,"threads_started":1}
I20260619 12:32:37.622820 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:37.616656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44504 (local address 127.0.98.2:40917)
0619 12:32:37.617047 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:37.617066 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:37.617147 (+ 81us) server_negotiation.cc:407] Connection header received
0619 12:32:37.617254 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:37.617261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:37.617549 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:37.617724 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:32:37.618203 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.619516 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.620576 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.621649 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.621745 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:37.622573 (+ 828us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:37.622617 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 12:32:37.622690 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":178,"threads_started":1}
I20260619 12:32:37.625931 14774 tablet_bootstrap.cc:492] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a: No bootstrap required, opened a new log
I20260619 12:32:37.625593 14954 tablet_service.cc:1467] Processing CreateTablet for tablet 00a03174fd07450f8307b41e01a0f109 (DEFAULT_TABLE table=test-table [id=cd0d682a125d4a3e941e09649afc60a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:32:37.626022 14774 ts_tablet_manager.cc:1397] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 12:32:37.625814 15086 tablet_service.cc:1467] Processing CreateTablet for tablet 00a03174fd07450f8307b41e01a0f109 (DEFAULT_TABLE table=test-table [id=cd0d682a125d4a3e941e09649afc60a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:32:37.626567 14954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a03174fd07450f8307b41e01a0f109. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:32:37.626574 14774 raft_consensus.cc:348] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260619 12:32:37.626703 14774 raft_consensus.cc:374] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:32:37.626741 14774 raft_consensus.cc:729] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a029cb92664b639ed82b023679810a, State: Initialized, Role: FOLLOWER
I20260619 12:32:37.626902 15086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a03174fd07450f8307b41e01a0f109. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:32:37.627187 14774 consensus_queue.cc:260] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260619 12:32:37.627677 14774 ts_tablet_manager.cc:1428] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 12:32:37.633522 15165 tablet_bootstrap.cc:492] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f: Bootstrap starting.
I20260619 12:32:37.634586 15166 tablet_bootstrap.cc:492] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133: Bootstrap starting.
I20260619 12:32:37.636034 15165 tablet_bootstrap.cc:654] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f: Neither blocks nor log segments found. Creating new log.
I20260619 12:32:37.636755 15165 log.cc:824] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:37.636857 15166 tablet_bootstrap.cc:654] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133: Neither blocks nor log segments found. Creating new log.
I20260619 12:32:37.637888 15166 log.cc:824] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:37.638876 15165 tablet_bootstrap.cc:492] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f: No bootstrap required, opened a new log
I20260619 12:32:37.639068 15165 ts_tablet_manager.cc:1397] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260619 12:32:37.640038 15166 tablet_bootstrap.cc:492] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133: No bootstrap required, opened a new log
I20260619 12:32:37.640237 15166 ts_tablet_manager.cc:1397] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:32:37.642026 15165 raft_consensus.cc:348] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260619 12:32:37.642313 15165 raft_consensus.cc:374] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:32:37.642373 15165 raft_consensus.cc:729] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a574e055cc414b6fbadfdb5c7290082f, State: Initialized, Role: FOLLOWER
I20260619 12:32:37.642814 15165 consensus_queue.cc:260] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260619 12:32:37.643491 15020 heartbeater.cc:503] Master 127.0.98.62:45791 was elected leader, sending a full tablet report...
I20260619 12:32:37.643733 15165 ts_tablet_manager.cc:1428] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260619 12:32:37.644055 15166 raft_consensus.cc:348] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260619 12:32:37.644315 15166 raft_consensus.cc:374] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:32:37.644379 15166 raft_consensus.cc:729] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7671644a7cbd4d3cb9fc7f6b44c7e133, State: Initialized, Role: FOLLOWER
I20260619 12:32:37.644824 15166 consensus_queue.cc:260] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260619 12:32:37.645501 15152 heartbeater.cc:503] Master 127.0.98.62:45791 was elected leader, sending a full tablet report...
I20260619 12:32:37.645925 15166 ts_tablet_manager.cc:1428] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
W20260619 12:32:37.679579 14883 tablet.cc:2369] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:32:37.830552 15153 tablet.cc:2369] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:32:37.880163 15170 raft_consensus.cc:482] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:32:37.880436 15170 raft_consensus.cc:504] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260619 12:32:37.881821 15170 leader_election.cc:288] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a574e055cc414b6fbadfdb5c7290082f (127.0.98.2:40917), 72a029cb92664b639ed82b023679810a (127.0.98.1:40125)
I20260619 12:32:37.885880 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:37.882250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51359 (local address 127.0.98.2:40917)
0619 12:32:37.882390 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:37.882395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:37.882655 (+ 260us) server_negotiation.cc:407] Connection header received
0619 12:32:37.882808 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:37.882812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:37.882867 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:37.882952 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:32:37.883224 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.883747 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.884802 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.885262 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.885379 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:37.885442 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:37.885647 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 12:32:37.885732 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:32:37.885950 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:37.882128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40917 (local address 127.0.98.3:51359)
0619 12:32:37.882548 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:37.882562 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:37.882678 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:37.882960 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:37.882963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:37.882978 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:32:37.883114 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:37.883119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.883858 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.883864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:37.884682 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:37.884691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.885384 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.885388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:37.885484 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:37.885539 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 12:32:37.885633 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 12:32:37.885725 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":103,"thread_start_us":106,"threads_started":1}
I20260619 12:32:37.885896 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:37.882237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58449 (local address 127.0.98.1:40125)
0619 12:32:37.882549 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:37.882553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:37.882566 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:32:37.882616 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:37.882618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:37.882668 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:37.882806 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:32:37.883087 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.883611 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.884763 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.885283 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.885404 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:37.885573 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:37.885666 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 12:32:37.885742 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":75,"threads_started":1}
W20260619 12:32:37.886050 15021 tablet.cc:2369] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:32:37.886451 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:37.882128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40125 (local address 127.0.98.3:58449)
0619 12:32:37.882346 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:37.882362 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:37.882467 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:37.882833 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:37.882837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:37.882854 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:32:37.882973 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:37.882979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.883734 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.883739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:37.884638 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:37.884647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.885414 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.885417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:37.885539 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:37.885566 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 12:32:37.885650 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 12:32:37.885732 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":55}
I20260619 12:32:37.886951 14974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a03174fd07450f8307b41e01a0f109" candidate_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a574e055cc414b6fbadfdb5c7290082f" is_pre_election: true
I20260619 12:32:37.887152 14836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a03174fd07450f8307b41e01a0f109" candidate_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72a029cb92664b639ed82b023679810a" is_pre_election: true
I20260619 12:32:37.887269 14974 raft_consensus.cc:2454] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7671644a7cbd4d3cb9fc7f6b44c7e133 in term 0.
I20260619 12:32:37.887441 14836 raft_consensus.cc:2454] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7671644a7cbd4d3cb9fc7f6b44c7e133 in term 0.
I20260619 12:32:37.887814 15041 leader_election.cc:302] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 72a029cb92664b639ed82b023679810a, 7671644a7cbd4d3cb9fc7f6b44c7e133, a574e055cc414b6fbadfdb5c7290082f; no voters:
I20260619 12:32:37.888109 15170 raft_consensus.cc:2781] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:32:37.888190 15170 raft_consensus.cc:482] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:32:37.888238 15170 raft_consensus.cc:3037] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:32:37.889323 15170 raft_consensus.cc:504] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260619 12:32:37.889693 15170 leader_election.cc:288] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [CANDIDATE]: Term 1 election: Requested vote from peers a574e055cc414b6fbadfdb5c7290082f (127.0.98.2:40917), 72a029cb92664b639ed82b023679810a (127.0.98.1:40125)
I20260619 12:32:37.890012 14974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a03174fd07450f8307b41e01a0f109" candidate_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a574e055cc414b6fbadfdb5c7290082f"
I20260619 12:32:37.890080 14836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a03174fd07450f8307b41e01a0f109" candidate_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72a029cb92664b639ed82b023679810a"
I20260619 12:32:37.890143 14974 raft_consensus.cc:3037] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:32:37.890197 14836 raft_consensus.cc:3037] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:32:37.891429 14836 raft_consensus.cc:2454] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7671644a7cbd4d3cb9fc7f6b44c7e133 in term 1.
I20260619 12:32:37.891429 14974 raft_consensus.cc:2454] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7671644a7cbd4d3cb9fc7f6b44c7e133 in term 1.
I20260619 12:32:37.891754 15039 leader_election.cc:302] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72a029cb92664b639ed82b023679810a, 7671644a7cbd4d3cb9fc7f6b44c7e133; no voters:
I20260619 12:32:37.891950 15170 raft_consensus.cc:2781] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:32:37.892150 15170 raft_consensus.cc:686] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 1 LEADER]: Becoming Leader. State: Replica: 7671644a7cbd4d3cb9fc7f6b44c7e133, State: Running, Role: LEADER
I20260619 12:32:37.892505 15170 consensus_queue.cc:237] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260619 12:32:37.895685 14694 catalog_manager.cc:5495] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7671644a7cbd4d3cb9fc7f6b44c7e133 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } health_report { overall_health: UNKNOWN } } }
I20260619 12:32:37.925865 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:37.921257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38034 (local address 127.0.98.3:33871)
0619 12:32:37.921396 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:37.921400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:37.921413 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:32:37.921504 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:37.921506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:37.921560 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:37.921648 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:32:37.921953 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.922528 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.923627 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:37.923869 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:37.923977 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:37.924346 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:32:37.924916 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:32:37.925064 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:37.925570 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 12:32:37.925647 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:32:37.937268 14836 raft_consensus.cc:1261] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Refusing update from remote peer 7671644a7cbd4d3cb9fc7f6b44c7e133: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:32:37.937418 14974 raft_consensus.cc:1261] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f [term 1 FOLLOWER]: Refusing update from remote peer 7671644a7cbd4d3cb9fc7f6b44c7e133: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:32:37.938058 15170 consensus_queue.cc:1035] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:32:37.938273 15173 consensus_queue.cc:1035] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:32:37.945261 15176 mvcc.cc:205] Tried to move back new op lower bound from 7298549178103345152 to 7298549177930526720. Current Snapshot: MvccSnapshot[applied={T|T < 7298549178103345152}]
I20260619 12:32:37.946460 15179 mvcc.cc:205] Tried to move back new op lower bound from 7298549178103345152 to 7298549177930526720. Current Snapshot: MvccSnapshot[applied={T|T < 7298549178103345152}]
I20260619 12:32:37.962327 14693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:32:37.962509 14693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:32:37.965268 14693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7: Sending DeleteTablet for 3 replicas of tablet 00a03174fd07450f8307b41e01a0f109
I20260619 12:32:37.966187 14816 tablet_service.cc:1514] Processing DeleteTablet for tablet 00a03174fd07450f8307b41e01a0f109 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:32:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37478
I20260619 12:32:37.966747 15086 tablet_service.cc:1514] Processing DeleteTablet for tablet 00a03174fd07450f8307b41e01a0f109 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:32:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38032
I20260619 12:32:37.967128 14954 tablet_service.cc:1514] Processing DeleteTablet for tablet 00a03174fd07450f8307b41e01a0f109 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:32:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44504
I20260619 12:32:37.967798 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 14732
I20260619 12:32:37.970278 15187 tablet_replica.cc:330] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133: stopping tablet replica
I20260619 12:32:37.970636 15187 raft_consensus.cc:2229] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:32:37.971092 15187 raft_consensus.cc:2258] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:32:37.972708 15187 ts_tablet_manager.cc:1905] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:32:37.973630 15188 tablet_replica.cc:330] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f: stopping tablet replica
I20260619 12:32:37.973951 15188 raft_consensus.cc:2229] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:32:37.974287 15188 raft_consensus.cc:2258] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:32:37.975638 15187 ts_tablet_manager.cc:1918] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:32:37.975756 15187 log.cc:1196] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/00a03174fd07450f8307b41e01a0f109
I20260619 12:32:37.975853 15188 ts_tablet_manager.cc:1905] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:32:37.976121 15187 ts_tablet_manager.cc:1939] T 00a03174fd07450f8307b41e01a0f109 P 7671644a7cbd4d3cb9fc7f6b44c7e133: Deleting consensus metadata
I20260619 12:32:37.977380 14679 catalog_manager.cc:4841] TS 7671644a7cbd4d3cb9fc7f6b44c7e133 (127.0.98.3:33871): tablet 00a03174fd07450f8307b41e01a0f109 (table test-table [id=cd0d682a125d4a3e941e09649afc60a6]) successfully deleted
I20260619 12:32:37.978374 15188 ts_tablet_manager.cc:1918] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:32:37.978492 15188 log.cc:1196] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/00a03174fd07450f8307b41e01a0f109
I20260619 12:32:37.978775 15188 ts_tablet_manager.cc:1939] T 00a03174fd07450f8307b41e01a0f109 P a574e055cc414b6fbadfdb5c7290082f: Deleting consensus metadata
W20260619 12:32:37.979727 14678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40125 (error 108)
I20260619 12:32:37.980334 14680 catalog_manager.cc:4841] TS a574e055cc414b6fbadfdb5c7290082f (127.0.98.2:40917): tablet 00a03174fd07450f8307b41e01a0f109 (table test-table [id=cd0d682a125d4a3e941e09649afc60a6]) successfully deleted
W20260619 12:32:37.980934 15161 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:32:37.980465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40125 (local address 127.0.0.1:37494)
0619 12:32:37.980632 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:37.980653 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:37.980722 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:37.980857 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40125: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":93}
W20260619 12:32:37.981145 14678 catalog_manager.cc:4584] TS 72a029cb92664b639ed82b023679810a (127.0.98.1:40125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 00a03174fd07450f8307b41e01a0f109: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40125: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 12:32:37.981221 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 14886
I20260619 12:32:37.987998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 15023
I20260619 12:32:37.994342 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 14661
I20260619 12:32:38.000958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45791
--webserver_interface=127.0.98.62
--webserver_port=45085
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:32:38.111860 15190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:38.112128 15190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:38.112169 15190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:38.116317 15190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:32:38.116386 15190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:38.116406 15190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:38.116452 15190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:32:38.116487 15190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:32:38.122033 15190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:38.123222 15190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:38.124264 15190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:38.132557 15195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:38.132570 15198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:38.132740 15196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:38.132946 15190 server_base.cc:1034] running on GCE node
I20260619 12:32:38.133545 15190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:38.134621 15190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:38.135784 15190 hybrid_clock.cc:648] HybridClock initialized: now 1781872358135771 us; error 26 us; skew 500 ppm
Jun 19 12:32:38 dist-test-slave-fg3h krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:38.138540 15190 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:32:38.139429 15190 webserver.cc:460] Webserver started at http://127.0.98.62:45085/ using document root <none> and password file <none>
I20260619 12:32:38.139930 15190 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:38.139987 15190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:38.143679 15190 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:32:38.145673 15205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:38.146880 15190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:32:38.147019 15190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "215ae7f9577a4b548d66e18c413d43f7"
format_stamp: "Formatted at 2026-06-19 12:32:35 on dist-test-slave-fg3h"
server_key: "cdd7055645a1c0f7cf95fc10d6ae0a05"
server_key_iv: "e3dd887944ea267ab1a9e647ad54b5e6"
server_key_version: "encryptionkey@0"
I20260619 12:32:38.147557 15190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:38.160715 15190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:38.163738 15190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:38.163940 15190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:38.170948 15190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45791
I20260619 12:32:38.171238 15257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45791 every 8 connection(s)
I20260619 12:32:38.172014 15190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:32:38.176252 15258 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:32:38.177151 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 15190
I20260619 12:32:38.177606 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40125
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45791
--webserver_port=34381
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:32:38.178205 15258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7: Bootstrap starting.
I20260619 12:32:38.195559 15258 log.cc:824] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:38.206710 15258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:32:38.207101 15258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7: Bootstrap complete.
I20260619 12:32:38.211241 15258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215ae7f9577a4b548d66e18c413d43f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260619 12:32:38.211686 15258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 215ae7f9577a4b548d66e18c413d43f7, State: Initialized, Role: FOLLOWER
I20260619 12:32:38.212224 15258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215ae7f9577a4b548d66e18c413d43f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260619 12:32:38.212343 15258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:32:38.212427 15258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:32:38.212548 15258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 2 FOLLOWER]: Advancing to term 3
I20260619 12:32:38.214493 15258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215ae7f9577a4b548d66e18c413d43f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260619 12:32:38.214835 15258 leader_election.cc:302] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 215ae7f9577a4b548d66e18c413d43f7; no voters:
I20260619 12:32:38.215145 15258 leader_election.cc:288] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 12:32:38.215292 15262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 12:32:38.215526 15262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [term 3 LEADER]: Becoming Leader. State: Replica: 215ae7f9577a4b548d66e18c413d43f7, State: Running, Role: LEADER
I20260619 12:32:38.215831 15262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215ae7f9577a4b548d66e18c413d43f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } }
I20260619 12:32:38.216398 15258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:32:38.217459 15263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "215ae7f9577a4b548d66e18c413d43f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215ae7f9577a4b548d66e18c413d43f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } } }
I20260619 12:32:38.217429 15264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 215ae7f9577a4b548d66e18c413d43f7. Latest consensus state: current_term: 3 leader_uuid: "215ae7f9577a4b548d66e18c413d43f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215ae7f9577a4b548d66e18c413d43f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45791 } } }
I20260619 12:32:38.217631 15263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [sys.catalog]: This master's current role is: LEADER
I20260619 12:32:38.217633 15264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7 [sys.catalog]: This master's current role is: LEADER
I20260619 12:32:38.222064 15277 catalog_manager.cc:1259] Loaded cluster ID: d0c48f9743a04229a1aa67e468b02c6f
I20260619 12:32:38.222136 15277 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7: loading cluster ID for follower catalog manager: success
W20260619 12:32:38.223210 15277 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 215ae7f9577a4b548d66e18c413d43f7: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260619 12:32:38.223285 15277 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260619 12:32:38.223346 15271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:32:38.226472 15271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:32:38.226951 15271 catalog_manager.cc:1259] Loaded cluster ID: d0c48f9743a04229a1aa67e468b02c6f
I20260619 12:32:38.227012 15271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 12:32:38.227625 15271 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f49e81ab9ad google::LogMessage::Fail() at ??:0
@ 0x7f49e81ae88a google::LogMessage::SendToLog() at ??:0
@ 0x7f49e81ab4de google::LogMessage::Flush() at ??:0
@ 0x7f49e81aca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f49f2aeafdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f49f2aebd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f49f2ae8579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f49f2b2c640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f49ea3fd3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f49ea3fdc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f49ea3ff3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f49ea3ee512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f49eba326db start_thread at ??:0
@ 0x7f49e6ca961f clone at ??:0
W20260619 12:32:38.310138 15260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:38.310462 15260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:38.310556 15260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:38.314751 15260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:38.314837 15260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:38.314924 15260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:32:38.320362 15260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34381
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:38.321617 15260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:38.322744 15260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:38.332651 15300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:38.332686 15299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:38.332655 15302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:38.333534 15260 server_base.cc:1034] running on GCE node
I20260619 12:32:38.333979 15260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:38.334674 15260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:38.335855 15260 hybrid_clock.cc:648] HybridClock initialized: now 1781872358335813 us; error 59 us; skew 500 ppm
Jun 19 12:32:38 dist-test-slave-fg3h krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:38.338763 15260 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:32:38.339771 15260 webserver.cc:460] Webserver started at http://127.0.98.1:34381/ using document root <none> and password file <none>
I20260619 12:32:38.340368 15260 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:38.340435 15260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:38.344516 15260 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:32:38.346925 15309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:38.348450 15260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:32:38.348611 15260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "72a029cb92664b639ed82b023679810a"
format_stamp: "Formatted at 2026-06-19 12:32:35 on dist-test-slave-fg3h"
server_key: "fb0a68cfadb919ef0ccf419a2efa575f"
server_key_iv: "65edf6272a95d181179cd15575505ec1"
server_key_version: "encryptionkey@0"
I20260619 12:32:38.349274 15260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:38.363852 15260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:38.367415 15260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:38.367663 15260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:38.368301 15260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:32:38.369516 15316 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 12:32:38.375212 15260 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 12:32:38.375310 15260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 12:32:38.375407 15260 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 12:32:38.377209 15260 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 12:32:38.377278 15260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260619 12:32:38.377561 15316 tablet_bootstrap.cc:492] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a: Bootstrap starting.
I20260619 12:32:38.388135 15260 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40125
I20260619 12:32:38.388168 15423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40125 every 8 connection(s)
I20260619 12:32:38.389212 15260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:32:38.394759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 15260
I20260619 12:32:38.395210 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40917
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45791
--webserver_port=35629
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:32:38.399148 15316 log.cc:824] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:38.406178 15316 tablet_bootstrap.cc:492] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:32:38.406550 15316 tablet_bootstrap.cc:492] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a: Bootstrap complete.
I20260619 12:32:38.407639 15316 ts_tablet_manager.cc:1397] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.014s
I20260619 12:32:38.410060 15316 raft_consensus.cc:348] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260619 12:32:38.410465 15316 raft_consensus.cc:729] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a029cb92664b639ed82b023679810a, State: Initialized, Role: FOLLOWER
I20260619 12:32:38.411003 15316 consensus_queue.cc:260] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260619 12:32:38.411978 15316 ts_tablet_manager.cc:1428] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
W20260619 12:32:38.478197 15426 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:32:38.391179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:60609)
0619 12:32:38.391676 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:38.391731 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:38.392571 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:38.477237 (+ 84666us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":147,"threads_started":1}
W20260619 12:32:38.478816 15424 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45791 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45791: Client connection negotiation failed: client connection to 127.0.98.62:45791: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 12:32:38.479611 15426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:38.479333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:51469)
0619 12:32:38.479474 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:38.479549 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 12:32:38.480317 15426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:38.480078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:50557)
0619 12:32:38.480195 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:38.480255 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 12:32:38.480515 15424 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:32:38.525520 15427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:38.525780 15427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:38.525861 15427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:38.529922 15427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:38.529978 15427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:38.530082 15427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:32:38.535264 15427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35629
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:38.536299 15427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:38.537322 15427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:38.547000 15435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:38.547006 15437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:38.546993 15434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:38.547518 15427 server_base.cc:1034] running on GCE node
I20260619 12:32:38.547935 15427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:38.548823 15427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:38.550050 15427 hybrid_clock.cc:648] HybridClock initialized: now 1781872358550010 us; error 51 us; skew 500 ppm
Jun 19 12:32:38 dist-test-slave-fg3h krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:38.552969 15427 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:32:38.554018 15427 webserver.cc:460] Webserver started at http://127.0.98.2:35629/ using document root <none> and password file <none>
I20260619 12:32:38.554607 15427 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:38.554670 15427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:38.558480 15427 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:32:38.560496 15444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:38.561777 15427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:32:38.561872 15427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "a574e055cc414b6fbadfdb5c7290082f"
format_stamp: "Formatted at 2026-06-19 12:32:36 on dist-test-slave-fg3h"
server_key: "7e7ca55982dbe12113a829b812879218"
server_key_iv: "b223b659fd99ef04953b38a2951b97f8"
server_key_version: "encryptionkey@0"
I20260619 12:32:38.562389 15427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:38.587965 15427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:38.590869 15427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:38.591025 15427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:38.591557 15427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:32:38.592301 15427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:32:38.592347 15427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:38.592386 15427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:32:38.592408 15427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:38.601737 15427 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40917
I20260619 12:32:38.601759 15557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40917 every 8 connection(s)
I20260619 12:32:38.602720 15427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:32:38.604123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 15427
I20260619 12:32:38.604498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33871
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45791
--webserver_port=40367
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:32:38.606884 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:38.605000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:46531)
0619 12:32:38.605952 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:38.606111 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":561,"threads_started":1}
W20260619 12:32:38.607440 15558 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45791 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45791: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
I20260619 12:32:38.608031 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:38.607812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:44907)
0619 12:32:38.607917 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:38.607965 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 12:32:38.608726 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:38.608542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:41751)
0619 12:32:38.608640 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:38.608679 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 12:32:38.608960 15558 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:32:38.716325 15560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:38.716595 15560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:38.716648 15560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:38.720806 15560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:38.720870 15560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:38.720930 15560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:32:38.726364 15560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40367
--tserver_master_addrs=127.0.98.62:45791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:38.727458 15560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:38.728489 15560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:32:38.738521 15560 server_base.cc:1034] running on GCE node
W20260619 12:32:38.738471 15569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:38.738474 15567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:38.738512 15566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:38.739151 15560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:38.739827 15560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:38.741009 15560 hybrid_clock.cc:648] HybridClock initialized: now 1781872358740978 us; error 41 us; skew 500 ppm
Jun 19 12:32:38 dist-test-slave-fg3h krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:38.743845 15560 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:32:38.744674 15560 webserver.cc:460] Webserver started at http://127.0.98.3:40367/ using document root <none> and password file <none>
I20260619 12:32:38.745270 15560 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:38.745337 15560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:38.749436 15560 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:32:38.751530 15576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:38.752928 15560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:32:38.753134 15560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133"
format_stamp: "Formatted at 2026-06-19 12:32:36 on dist-test-slave-fg3h"
server_key: "f146a8ca929b7b10a2c866ade53d0eac"
server_key_iv: "73f50742f64b870c5fb95dfb6788353d"
server_key_version: "encryptionkey@0"
I20260619 12:32:38.753731 15560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:38.772845 15560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:38.776047 15560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:38.776252 15560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:38.776898 15560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:32:38.777781 15560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:32:38.777828 15560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:38.777901 15560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:32:38.777935 15560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:38.786957 15560 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33871
I20260619 12:32:38.787073 15689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33871 every 8 connection(s)
I20260619 12:32:38.788033 15560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:32:38.791145 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 15560
I20260619 12:32:38.792174 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:38.790223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:51979)
0619 12:32:38.790688 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:38.791389 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":114,"threads_started":1}
W20260619 12:32:38.792632 15690 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45791 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45791: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
I20260619 12:32:38.793274 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:38.792999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:36073)
0619 12:32:38.793175 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:38.793221 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 12:32:38.793859 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:38.793678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:47327)
0619 12:32:38.793773 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:38.793812 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 12:32:38.794049 15690 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 12:32:39.481627 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:39.481179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:49869)
0619 12:32:39.481444 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:39.481510 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
I20260619 12:32:39.609916 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:39.609541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:45455)
0619 12:32:39.609773 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:39.609825 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":73,"threads_started":1}
I20260619 12:32:39.795077 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:39.794641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:49847)
0619 12:32:39.794912 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:39.794972 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":77,"threads_started":1}
I20260619 12:32:40.117955 15697 raft_consensus.cc:482] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:32:40.118197 15697 raft_consensus.cc:504] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260619 12:32:40.119539 15697 leader_election.cc:288] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7671644a7cbd4d3cb9fc7f6b44c7e133 (127.0.98.3:33871), a574e055cc414b6fbadfdb5c7290082f (127.0.98.2:40917)
Jun 19 12:32:40 dist-test-slave-fg3h krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 19 12:32:40 dist-test-slave-fg3h krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260619 12:32:40.134303 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:40.119776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33871 (local address 127.0.98.1:39769)
0619 12:32:40.120089 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:40.120131 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:40.120249 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:40.121893 (+ 1644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:40.121896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:40.122490 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:40.122960 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:40.122977 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:40.124271 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:40.124275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:40.125378 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:40.125388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:40.126194 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:40.126197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:40.126372 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:40.127095 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:40.127123 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:40.128796 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:40.131436 (+ 2640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:40.131446 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:40.131460 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:40.131792 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:40.132240 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:40.132242 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:40.132243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:40.132430 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:40.133262 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:40.133268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:40.133541 (+ 273us) client_negotiation.cc:769] Sending connection context
0619 12:32:40.133706 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 12:32:40.134075 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":88,"spinlock_wait_cycles":18048,"thread_start_us":75,"threads_started":1}
I20260619 12:32:40.134320 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:40.119923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53433 (local address 127.0.98.2:40917)
0619 12:32:40.120320 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:40.120332 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:40.120368 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:32:40.121039 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:40.121044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:40.121504 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:40.121789 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:40.123143 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:40.124116 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:40.125557 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:40.126032 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:40.126203 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:40.129426 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:40.129433 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:40.129450 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:40.129488 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:40.131818 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:40.132362 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:40.132367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:40.132372 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:40.132505 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:40.132863 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:40.132865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:40.132866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:40.133332 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:40.133457 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:40.133879 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 12:32:40.134135 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":110,"threads_started":1}
I20260619 12:32:40.134383 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:40.119923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39769 (local address 127.0.98.3:33871)
0619 12:32:40.120510 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:40.120529 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:40.120564 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:32:40.121267 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:40.121273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:40.121624 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:40.121980 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:40.123146 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:40.124118 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:40.125506 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:40.126043 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:40.126258 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:40.129059 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:40.129065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:40.129078 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:40.129173 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:40.131315 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:40.131945 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:40.131952 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:40.131958 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:40.132135 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:40.132562 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:40.132565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:40.132566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:40.133152 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:40.133259 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:40.133914 (+ 655us) server_negotiation.cc:299] Negotiation successful
0619 12:32:40.134188 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":92,"threads_started":1}
I20260619 12:32:40.134992 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:40.119776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40917 (local address 127.0.98.1:53433)
0619 12:32:40.120198 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:40.120211 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:40.120282 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:40.121887 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:40.121894 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:40.122475 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:40.122959 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:40.122975 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:40.124285 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:40.124288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:40.125378 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:40.125388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:40.126167 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:40.126171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:40.126364 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:40.127093 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:40.127112 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:40.129194 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:40.131931 (+ 2737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:40.131949 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:40.131951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:40.132248 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:40.132673 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:40.132676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:40.132678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:40.132764 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:40.133455 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:40.133457 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:40.133549 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 12:32:40.133749 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 12:32:40.134176 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":384,"spinlock_wait_cycles":5760,"thread_start_us":60,"threads_started":1}
W20260619 12:32:40.137029 15311 leader_election.cc:341] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7671644a7cbd4d3cb9fc7f6b44c7e133 (127.0.98.3:33871): Not found: Tablet not found: 00a03174fd07450f8307b41e01a0f109
W20260619 12:32:40.137247 15312 leader_election.cc:341] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a574e055cc414b6fbadfdb5c7290082f (127.0.98.2:40917): Not found: Tablet not found: 00a03174fd07450f8307b41e01a0f109
I20260619 12:32:40.137351 15312 leader_election.cc:302] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72a029cb92664b639ed82b023679810a; no voters: 7671644a7cbd4d3cb9fc7f6b44c7e133, a574e055cc414b6fbadfdb5c7290082f
I20260619 12:32:40.137647 15697 raft_consensus.cc:2726] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:32:40.482673 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:40.482384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:33295)
0619 12:32:40.482533 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:40.482590 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 12:32:40.611248 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:40.610754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:39627)
0619 12:32:40.611070 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:40.611132 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":99,"threads_started":1}
I20260619 12:32:40.796370 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:40.795880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:43937)
0619 12:32:40.796157 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:40.796218 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":76,"threads_started":1}
I20260619 12:32:41.483932 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:41.483512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:41239)
0619 12:32:41.483771 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:41.483833 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
I20260619 12:32:41.612536 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:41.612126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:51571)
0619 12:32:41.612384 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:41.612438 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":77,"threads_started":1}
I20260619 12:32:41.704069 15706 raft_consensus.cc:482] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:32:41.704201 15706 raft_consensus.cc:504] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260619 12:32:41.704584 15706 leader_election.cc:288] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7671644a7cbd4d3cb9fc7f6b44c7e133 (127.0.98.3:33871), a574e055cc414b6fbadfdb5c7290082f (127.0.98.2:40917)
W20260619 12:32:41.705489 15312 leader_election.cc:341] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a574e055cc414b6fbadfdb5c7290082f (127.0.98.2:40917): Not found: Tablet not found: 00a03174fd07450f8307b41e01a0f109
W20260619 12:32:41.705605 15311 leader_election.cc:341] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7671644a7cbd4d3cb9fc7f6b44c7e133 (127.0.98.3:33871): Not found: Tablet not found: 00a03174fd07450f8307b41e01a0f109
I20260619 12:32:41.705792 15311 leader_election.cc:302] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72a029cb92664b639ed82b023679810a; no voters: 7671644a7cbd4d3cb9fc7f6b44c7e133, a574e055cc414b6fbadfdb5c7290082f
I20260619 12:32:41.705986 15706 raft_consensus.cc:2726] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:32:41.797639 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:41.797184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:53707)
0619 12:32:41.797480 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:41.797538 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":69,"threads_started":1}
I20260619 12:32:42.485257 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:42.484763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:52327)
0619 12:32:42.485036 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:42.485160 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":79,"threads_started":1}
I20260619 12:32:42.613859 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:42.613407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:42355)
0619 12:32:42.613706 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:42.613767 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":121,"threads_started":1}
I20260619 12:32:42.798889 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:42.798433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:49459)
0619 12:32:42.798737 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:42.798798 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":82,"threads_started":1}
I20260619 12:32:43.486646 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:43.486188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:33653)
0619 12:32:43.486485 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:43.486550 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":77,"threads_started":1}
I20260619 12:32:43.615161 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:43.614729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:45915)
0619 12:32:43.615000 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:43.615063 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260619 12:32:43.800128 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:43.799707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:54781)
0619 12:32:43.799973 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:43.800035 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
I20260619 12:32:44.487807 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:44.487412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:58163)
0619 12:32:44.487660 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:44.487714 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
I20260619 12:32:44.616451 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:44.616043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:60071)
0619 12:32:44.616292 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:44.616354 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20260619 12:32:44.675451 15716 raft_consensus.cc:482] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:32:44.675611 15716 raft_consensus.cc:504] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260619 12:32:44.676098 15716 leader_election.cc:288] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7671644a7cbd4d3cb9fc7f6b44c7e133 (127.0.98.3:33871), a574e055cc414b6fbadfdb5c7290082f (127.0.98.2:40917)
W20260619 12:32:44.676865 15312 leader_election.cc:341] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a574e055cc414b6fbadfdb5c7290082f (127.0.98.2:40917): Not found: Tablet not found: 00a03174fd07450f8307b41e01a0f109
W20260619 12:32:44.677002 15311 leader_election.cc:341] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7671644a7cbd4d3cb9fc7f6b44c7e133 (127.0.98.3:33871): Not found: Tablet not found: 00a03174fd07450f8307b41e01a0f109
I20260619 12:32:44.677145 15311 leader_election.cc:302] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72a029cb92664b639ed82b023679810a; no voters: 7671644a7cbd4d3cb9fc7f6b44c7e133, a574e055cc414b6fbadfdb5c7290082f
I20260619 12:32:44.677281 15716 raft_consensus.cc:2726] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:32:44.801319 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:44.800834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:47253)
0619 12:32:44.801165 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:44.801219 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":71,"threads_started":1}
I20260619 12:32:45.488974 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:45.488597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:39401)
0619 12:32:45.488827 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:45.488880 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
I20260619 12:32:45.617756 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:45.617277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:38283)
0619 12:32:45.617597 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:45.617659 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":74,"threads_started":1}
I20260619 12:32:45.802862 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:45.802395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:34783)
0619 12:32:45.802709 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:45.802767 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":95,"threads_started":1}
I20260619 12:32:46.490226 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:46.489801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:53067)
0619 12:32:46.490071 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:46.490131 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":74,"threads_started":1}
I20260619 12:32:46.618999 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:46.618554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:39191)
0619 12:32:46.618845 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:46.618903 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":100,"threads_started":1}
I20260619 12:32:46.804044 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:46.803613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:33259)
0619 12:32:46.803886 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:46.803949 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
I20260619 12:32:47.491580 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:47.491101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:52453)
0619 12:32:47.491402 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:47.491467 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":94,"threads_started":1}
I20260619 12:32:47.620261 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:47.619803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:59363)
0619 12:32:47.620102 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:47.620163 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":79,"threads_started":1}
I20260619 12:32:47.805347 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:47.804823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:45463)
0619 12:32:47.805125 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:47.805197 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":84,"threads_started":1}
I20260619 12:32:47.932876 15727 raft_consensus.cc:482] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:32:47.933063 15727 raft_consensus.cc:504] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260619 12:32:47.933630 15727 leader_election.cc:288] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7671644a7cbd4d3cb9fc7f6b44c7e133 (127.0.98.3:33871), a574e055cc414b6fbadfdb5c7290082f (127.0.98.2:40917)
W20260619 12:32:47.934381 15311 leader_election.cc:341] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7671644a7cbd4d3cb9fc7f6b44c7e133 (127.0.98.3:33871): Not found: Tablet not found: 00a03174fd07450f8307b41e01a0f109
W20260619 12:32:47.934484 15312 leader_election.cc:341] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a574e055cc414b6fbadfdb5c7290082f (127.0.98.2:40917): Not found: Tablet not found: 00a03174fd07450f8307b41e01a0f109
I20260619 12:32:47.934577 15312 leader_election.cc:302] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72a029cb92664b639ed82b023679810a; no voters: 7671644a7cbd4d3cb9fc7f6b44c7e133, a574e055cc414b6fbadfdb5c7290082f
I20260619 12:32:47.934746 15727 raft_consensus.cc:2726] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:32:48.493072 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:48.492569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:35921)
0619 12:32:48.492892 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:48.492972 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":135,"threads_started":1}
I20260619 12:32:48.621796 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:48.621295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:49535)
0619 12:32:48.621603 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:48.621699 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":76,"threads_started":1}
I20260619 12:32:48.806712 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:48.806279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:45769)
0619 12:32:48.806554 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:48.806615 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260619 12:32:49.494370 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:49.493909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:41331)
0619 12:32:49.494204 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:49.494267 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":85,"threads_started":1}
I20260619 12:32:49.622953 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:49.622539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:60815)
0619 12:32:49.622783 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:49.622858 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260619 12:32:49.807835 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:49.807421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:39265)
0619 12:32:49.807677 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:49.807740 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
I20260619 12:32:50.495802 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:50.495350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:58959)
0619 12:32:50.495643 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:50.495708 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":83,"threads_started":1}
I20260619 12:32:50.624298 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:50.623827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:33009)
0619 12:32:50.624113 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:50.624183 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":72,"threads_started":1}
I20260619 12:32:50.809181 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:50.808621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:42591)
0619 12:32:50.808949 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:50.809008 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":128,"threads_started":1}
I20260619 12:32:51.497179 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:51.496644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:43761)
0619 12:32:51.496967 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:51.497028 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":98,"threads_started":1}
I20260619 12:32:51.625605 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:51.625192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:33549)
0619 12:32:51.625452 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:51.625507 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":89,"threads_started":1}
I20260619 12:32:51.810351 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:51.809938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:33755)
0619 12:32:51.810195 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:51.810246 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":105,"threads_started":1}
I20260619 12:32:52.498396 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:52.497977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:40361)
0619 12:32:52.498235 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:52.498296 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
I20260619 12:32:52.626955 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:52.626461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:37553)
0619 12:32:52.626786 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:52.626851 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":99,"threads_started":1}
I20260619 12:32:52.671046 15742 raft_consensus.cc:482] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:32:52.671206 15742 raft_consensus.cc:504] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7671644a7cbd4d3cb9fc7f6b44c7e133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33871 } } peers { permanent_uuid: "a574e055cc414b6fbadfdb5c7290082f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40917 } } peers { permanent_uuid: "72a029cb92664b639ed82b023679810a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40125 } }
I20260619 12:32:52.671702 15742 leader_election.cc:288] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7671644a7cbd4d3cb9fc7f6b44c7e133 (127.0.98.3:33871), a574e055cc414b6fbadfdb5c7290082f (127.0.98.2:40917)
W20260619 12:32:52.672482 15311 leader_election.cc:341] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7671644a7cbd4d3cb9fc7f6b44c7e133 (127.0.98.3:33871): Not found: Tablet not found: 00a03174fd07450f8307b41e01a0f109
W20260619 12:32:52.672608 15312 leader_election.cc:341] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a574e055cc414b6fbadfdb5c7290082f (127.0.98.2:40917): Not found: Tablet not found: 00a03174fd07450f8307b41e01a0f109
I20260619 12:32:52.672699 15312 leader_election.cc:302] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72a029cb92664b639ed82b023679810a; no voters: 7671644a7cbd4d3cb9fc7f6b44c7e133, a574e055cc414b6fbadfdb5c7290082f
I20260619 12:32:52.672853 15742 raft_consensus.cc:2726] T 00a03174fd07450f8307b41e01a0f109 P 72a029cb92664b639ed82b023679810a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 12:32:52.811597 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:52.811131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:47687)
0619 12:32:52.811434 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:52.811501 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":99,"threads_started":1}
I20260619 12:32:53.499818 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:53.499388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.1:53923)
0619 12:32:53.499669 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:53.499726 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":93,"threads_started":1}
I20260619 12:32:53.628378 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:53.627951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.2:45433)
0619 12:32:53.628227 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:53.628282 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":93,"threads_started":1}
I20260619 12:32:53.801565 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 15260
I20260619 12:32:53.807331 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 15427
I20260619 12:32:53.812994 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:53.812541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45791 (local address 127.0.98.3:53583)
0619 12:32:53.812842 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:53.812902 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
I20260619 12:32:53.813282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 15560
2026-06-19T12:32:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19638 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:32:53 dist-test-slave-fg3h krb5kdc[15749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:32:53 dist-test-slave-fg3h krb5kdc[15749](info): set up 2 sockets
Jun 19 12:32:53 dist-test-slave-fg3h krb5kdc[15749](info): commencing operation
krb5kdc: starting...
W20260619 12:32:54.885422 392 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:32:54 dist-test-slave-fg3h krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:32:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:32:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:32:55.012120 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36539
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:32:55.121793 15762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:55.122124 15762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:55.122207 15762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:55.126308 15762 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 12:32:55.126400 15762 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 12:32:55.126427 15762 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 12:32:55.126458 15762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:32:55.126523 15762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:55.126579 15762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:55.126618 15762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:32:55.126653 15762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:32:55.132088 15762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:55.133206 15762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:55.134259 15762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:55.142925 15767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:55.143023 15768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:55.142925 15770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:55.143182 15762 server_base.cc:1034] running on GCE node
I20260619 12:32:55.143852 15762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:55.144970 15762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:55.146159 15762 hybrid_clock.cc:648] HybridClock initialized: now 1781872375146146 us; error 28 us; skew 500 ppm
Jun 19 12:32:55 dist-test-slave-fg3h krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:55.148864 15762 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:32:55.149825 15762 webserver.cc:460] Webserver started at http://127.0.98.62:46343/ using document root <none> and password file <none>
I20260619 12:32:55.150386 15762 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:55.150478 15762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:55.150678 15762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:32:55.152643 15762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "7c0fd18d86c046059f47a525549b2cc2"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "647da8ea1f0ff2bfac75336f6731794e"
server_key_iv: "c23d1abc81daf8647a863a8307d14bac"
server_key_version: "encryptionkey@0"
I20260619 12:32:55.153249 15762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "7c0fd18d86c046059f47a525549b2cc2"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "647da8ea1f0ff2bfac75336f6731794e"
server_key_iv: "c23d1abc81daf8647a863a8307d14bac"
server_key_version: "encryptionkey@0"
I20260619 12:32:55.156775 15762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:32:55.159188 15777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:55.160420 15762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:32:55.160557 15762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "7c0fd18d86c046059f47a525549b2cc2"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "647da8ea1f0ff2bfac75336f6731794e"
server_key_iv: "c23d1abc81daf8647a863a8307d14bac"
server_key_version: "encryptionkey@0"
I20260619 12:32:55.160667 15762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:55.172936 15762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:55.175797 15762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:55.176009 15762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:55.182875 15762 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36539
I20260619 12:32:55.183163 15829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36539 every 8 connection(s)
I20260619 12:32:55.183861 15762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:32:55.186619 15830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:32:55.188416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 15762
I20260619 12:32:55.188515 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:32:55.188846 392 external_mini_cluster.cc:1428] Setting key 4e5782c03525d895865f19454d1b5364
I20260619 12:32:55.193246 15830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: Bootstrap starting.
I20260619 12:32:55.195773 15830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: Neither blocks nor log segments found. Creating new log.
I20260619 12:32:55.196684 15830 log.cc:824] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: Log is configured to *not* fsync() on all Append() calls
Jun 19 12:32:55 dist-test-slave-fg3h krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:32:55.198684 15830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: No bootstrap required, opened a new log
I20260619 12:32:55.202056 15830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0fd18d86c046059f47a525549b2cc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260619 12:32:55.202346 15830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:32:55.202435 15830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0fd18d86c046059f47a525549b2cc2, State: Initialized, Role: FOLLOWER
I20260619 12:32:55.202860 15830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0fd18d86c046059f47a525549b2cc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260619 12:32:55.203007 15830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:32:55.203092 15830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:32:55.203105 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:55.190302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40836 (local address 127.0.98.62:36539)
0619 12:32:55.190688 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:55.190714 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:55.190750 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:32:55.191399 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:55.191416 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:55.191727 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:55.192000 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:55.192907 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.193978 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.195137 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.195593 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.195706 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:55.198494 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:55.198498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:55.198511 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:55.198537 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:55.200209 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:55.200648 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:55.200660 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:55.200664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:55.200766 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:55.201052 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:55.201057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:55.201058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:55.201489 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:55.201698 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:55.201943 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 12:32:55.202209 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":131,"threads_started":1}
I20260619 12:32:55.203197 15830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:32:55.204277 15830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0fd18d86c046059f47a525549b2cc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260619 12:32:55.204615 15830 leader_election.cc:302] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c0fd18d86c046059f47a525549b2cc2; no voters:
I20260619 12:32:55.204909 15830 leader_election.cc:288] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:32:55.205053 15835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:32:55.205319 15835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 1 LEADER]: Becoming Leader. State: Replica: 7c0fd18d86c046059f47a525549b2cc2, State: Running, Role: LEADER
I20260619 12:32:55.205672 15835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0fd18d86c046059f47a525549b2cc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260619 12:32:55.206192 15830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:32:55.206786 15835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c0fd18d86c046059f47a525549b2cc2. Latest consensus state: current_term: 1 leader_uuid: "7c0fd18d86c046059f47a525549b2cc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0fd18d86c046059f47a525549b2cc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } } }
I20260619 12:32:55.206940 15835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [sys.catalog]: This master's current role is: LEADER
I20260619 12:32:55.206792 15837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c0fd18d86c046059f47a525549b2cc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0fd18d86c046059f47a525549b2cc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } } }
I20260619 12:32:55.207764 15837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [sys.catalog]: This master's current role is: LEADER
W20260619 12:32:55.210861 15850 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:32:55.210928 15850 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:32:55.211023 15844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:32:55.211900 15844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:32:55.217556 15844 catalog_manager.cc:1347] Generated new cluster ID: 45e497b2a82341568485c075dd2a4d85
I20260619 12:32:55.217628 15844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:32:55.237578 15844 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:32:55.238922 15844 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:32:55.249243 15844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: Generated new TSK 0
I20260619 12:32:55.250077 15844 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:32:55.282171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36539
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:32:55.393311 15856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:55.393620 15856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:55.393699 15856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:55.397758 15856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:55.397837 15856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:55.397933 15856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:32:55.403105 15856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:55.404165 15856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:55.405228 15856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:55.414984 15862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:55.414994 15864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:55.415058 15861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:55.415406 15856 server_base.cc:1034] running on GCE node
I20260619 12:32:55.415808 15856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:55.416428 15856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:55.417583 15856 hybrid_clock.cc:648] HybridClock initialized: now 1781872375417569 us; error 24 us; skew 500 ppm
Jun 19 12:32:55 dist-test-slave-fg3h krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:55.420265 15856 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:32:55.421239 15856 webserver.cc:460] Webserver started at http://127.0.98.1:42423/ using document root <none> and password file <none>
I20260619 12:32:55.421805 15856 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:55.421901 15856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:55.422134 15856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:32:55.424117 15856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "d44216947d9046ca9641bb826cfc5531"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "c9cf69287ca50f05a0b1f519dc6e7ef0"
server_key_iv: "440ec8a5df881798ffe2a32b745a31c0"
server_key_version: "encryptionkey@0"
I20260619 12:32:55.424701 15856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d44216947d9046ca9641bb826cfc5531"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "c9cf69287ca50f05a0b1f519dc6e7ef0"
server_key_iv: "440ec8a5df881798ffe2a32b745a31c0"
server_key_version: "encryptionkey@0"
I20260619 12:32:55.428526 15856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 12:32:55.431066 15871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:55.432271 15856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:32:55.432417 15856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "d44216947d9046ca9641bb826cfc5531"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "c9cf69287ca50f05a0b1f519dc6e7ef0"
server_key_iv: "440ec8a5df881798ffe2a32b745a31c0"
server_key_version: "encryptionkey@0"
I20260619 12:32:55.432533 15856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:55.451876 15856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:55.454916 15856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:55.455123 15856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:55.455715 15856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:32:55.456534 15856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:32:55.456606 15856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:55.456674 15856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:32:55.456723 15856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:55.465919 15856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38819
I20260619 12:32:55.465940 15984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38819 every 8 connection(s)
I20260619 12:32:55.466795 15856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:32:55.468782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 15856
I20260619 12:32:55.468927 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:32:55.469347 392 external_mini_cluster.cc:1428] Setting key e3e54302568f252f8a9bdf33f64454da
Jun 19 12:32:55 dist-test-slave-fg3h krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:32:55.482573 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:55.468543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56587 (local address 127.0.98.62:36539)
0619 12:32:55.468714 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:55.468720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:55.469955 (+ 1235us) server_negotiation.cc:407] Connection header received
0619 12:32:55.470842 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:55.470844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:55.470892 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:55.470956 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:55.472580 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.473491 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.475436 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.475702 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.475790 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:55.478806 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:55.478809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:55.478811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:55.478836 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:55.480387 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:55.480919 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:55.480923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:55.480925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:55.481044 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:55.481431 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:55.481435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:55.481437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:55.481627 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:55.481737 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:55.482251 (+ 514us) server_negotiation.cc:299] Negotiation successful
0619 12:32:55.482395 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:32:55.483568 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:55.468971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:56587)
0619 12:32:55.469807 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:55.469849 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:55.470634 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:55.471142 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:55.471153 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:55.471684 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:55.472399 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:55.472421 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.473662 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.473665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:55.475272 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:55.475289 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.475877 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.475882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:55.476122 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:55.476825 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:55.476851 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:55.478622 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:55.480514 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:55.480520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:55.480536 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:55.480801 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:55.481173 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:55.481176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:55.481178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:55.481325 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:55.481736 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:55.481742 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:55.482007 (+ 265us) client_negotiation.cc:769] Sending connection context
0619 12:32:55.482212 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 12:32:55.482539 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":120,"threads_started":1}
I20260619 12:32:55.484864 15985 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36539
I20260619 12:32:55.485167 15985 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:55.485672 15985 heartbeater.cc:511] Master 127.0.98.62:36539 requested a full tablet report, sending...
I20260619 12:32:55.487646 15794 ts_manager.cc:194] Registered new tserver with Master: d44216947d9046ca9641bb826cfc5531 (127.0.98.1:38819)
I20260619 12:32:55.489434 15794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:32:55.501025 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36539
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:32:55.614847 15990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:55.615154 15990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:55.615231 15990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:55.619364 15990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:55.619464 15990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:55.619557 15990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:32:55.624936 15990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:55.626071 15990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:55.627135 15990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:55.636621 15995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:55.636633 15996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:55.636633 15998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:55.636839 15990 server_base.cc:1034] running on GCE node
I20260619 12:32:55.637420 15990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:55.638028 15990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:55.639185 15990 hybrid_clock.cc:648] HybridClock initialized: now 1781872375639168 us; error 28 us; skew 500 ppm
Jun 19 12:32:55 dist-test-slave-fg3h krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:55.642055 15990 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:32:55.642876 15990 webserver.cc:460] Webserver started at http://127.0.98.2:40815/ using document root <none> and password file <none>
I20260619 12:32:55.643404 15990 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:55.643462 15990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:55.643622 15990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:32:55.645695 15990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "3875c93147994ff1b3b64768c373c049"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "bc4023187a69a8312bcdc67466eebcda"
server_key_iv: "cf3f5f9aeb75590c7e1694142d79b056"
server_key_version: "encryptionkey@0"
I20260619 12:32:55.646203 15990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3875c93147994ff1b3b64768c373c049"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "bc4023187a69a8312bcdc67466eebcda"
server_key_iv: "cf3f5f9aeb75590c7e1694142d79b056"
server_key_version: "encryptionkey@0"
I20260619 12:32:55.649955 15990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:32:55.652272 16005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:55.653569 15990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:32:55.653702 15990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "3875c93147994ff1b3b64768c373c049"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "bc4023187a69a8312bcdc67466eebcda"
server_key_iv: "cf3f5f9aeb75590c7e1694142d79b056"
server_key_version: "encryptionkey@0"
I20260619 12:32:55.653816 15990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:55.668857 15990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:55.672036 15990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:55.672279 15990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:55.672909 15990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:32:55.673844 15990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:32:55.673923 15990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:55.673995 15990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:32:55.674052 15990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:55.684607 15990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43239
I20260619 12:32:55.684659 16118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43239 every 8 connection(s)
I20260619 12:32:55.685739 15990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:32:55.687202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 15990
I20260619 12:32:55.687338 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:32:55.687657 392 external_mini_cluster.cc:1428] Setting key 966a09325043821b01e7ec5e4cc496f0
Jun 19 12:32:55 dist-test-slave-fg3h krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 12:32:55.702339 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:55.687785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56723 (local address 127.0.98.62:36539)
0619 12:32:55.687948 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:55.687953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:55.689222 (+ 1269us) server_negotiation.cc:407] Connection header received
0619 12:32:55.690154 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:55.690157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:55.690206 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:55.690284 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:55.691893 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.692474 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.694363 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.694581 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.694674 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:55.697946 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:55.697949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:55.697952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:55.697978 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:55.699726 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:55.700378 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:55.700385 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:55.700390 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:55.700487 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:55.700920 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:55.700923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:55.700924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:55.701166 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:55.701246 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:55.701968 (+ 722us) server_negotiation.cc:299] Negotiation successful
0619 12:32:55.702121 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 12:32:55.703509 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:55.688245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:56723)
0619 12:32:55.689038 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:55.689117 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:55.689950 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:55.690459 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:55.690466 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:55.691000 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:55.691706 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:55.691726 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.692619 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.692626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:55.694209 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:55.694225 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.694755 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.694760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:55.695001 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:55.695807 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:55.695828 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:55.697759 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:55.699859 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:55.699865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:55.699881 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:55.700223 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:55.700594 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:55.700599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:55.700601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:55.700811 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:55.701281 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:55.701291 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:55.701657 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 12:32:55.701889 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 12:32:55.702318 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":173,"threads_started":1}
I20260619 12:32:55.704871 16119 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36539
I20260619 12:32:55.705221 16119 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:55.705760 16119 heartbeater.cc:511] Master 127.0.98.62:36539 requested a full tablet report, sending...
I20260619 12:32:55.706955 15794 ts_manager.cc:194] Registered new tserver with Master: 3875c93147994ff1b3b64768c373c049 (127.0.98.2:43239)
I20260619 12:32:55.707821 15794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:32:55.716740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36539
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:32:55.829459 16124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:32:55.829733 16124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:32:55.829777 16124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:32:55.833861 16124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:32:55.833920 16124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:32:55.834012 16124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:32:55.839846 16124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:32:55.840960 16124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:32:55.842065 16124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:32:55.853057 16132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:32:55.853114 16130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:55.853338 16124 server_base.cc:1034] running on GCE node
W20260619 12:32:55.853111 16129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:32:55.853858 16124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:32:55.854553 16124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:32:55.855773 16124 hybrid_clock.cc:648] HybridClock initialized: now 1781872375855731 us; error 53 us; skew 500 ppm
Jun 19 12:32:55 dist-test-slave-fg3h krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:32:55.858975 16124 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:32:55.859812 16124 webserver.cc:460] Webserver started at http://127.0.98.3:40559/ using document root <none> and password file <none>
I20260619 12:32:55.860355 16124 fs_manager.cc:359] Metadata directory not provided
I20260619 12:32:55.860412 16124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:32:55.860576 16124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:32:55.862706 16124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "53c12ab3bbf34eda92d5f2477281e683"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "525b5e1ee93fdf8d4b4bacc398f2fed3"
server_key_iv: "12f0dce3f1276957870c44e1adad1d1d"
server_key_version: "encryptionkey@0"
I20260619 12:32:55.863233 16124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "53c12ab3bbf34eda92d5f2477281e683"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "525b5e1ee93fdf8d4b4bacc398f2fed3"
server_key_iv: "12f0dce3f1276957870c44e1adad1d1d"
server_key_version: "encryptionkey@0"
I20260619 12:32:55.867385 16124 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:32:55.869913 16139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:55.871299 16124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:32:55.871400 16124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "53c12ab3bbf34eda92d5f2477281e683"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "525b5e1ee93fdf8d4b4bacc398f2fed3"
server_key_iv: "12f0dce3f1276957870c44e1adad1d1d"
server_key_version: "encryptionkey@0"
I20260619 12:32:55.871492 16124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:32:55.881425 16124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:32:55.884590 16124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:32:55.884819 16124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:32:55.885456 16124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:32:55.886214 16124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:32:55.886267 16124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:55.886305 16124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:32:55.886327 16124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:32:55.895032 16124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44023
I20260619 12:32:55.895056 16252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44023 every 8 connection(s)
I20260619 12:32:55.895985 16124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:32:55.903249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 16124
I20260619 12:32:55.903429 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:32:55.903744 392 external_mini_cluster.cc:1428] Setting key 78717434c315f5a7616186e9b2d8d4f9
Jun 19 12:32:55 dist-test-slave-fg3h krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:32:55.909477 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:55.897792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36447 (local address 127.0.98.62:36539)
0619 12:32:55.897925 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:55.897928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:55.898968 (+ 1040us) server_negotiation.cc:407] Connection header received
0619 12:32:55.899775 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:55.899777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:55.899822 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:55.899901 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:55.901024 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.901630 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.902746 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.902975 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.903053 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:55.905844 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:55.905850 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:55.905856 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:55.905889 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:55.907286 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:55.908001 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:55.908003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:55.908005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:55.908078 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:55.908505 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:55.908507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:55.908508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:55.908675 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:55.908750 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:55.909263 (+ 513us) server_negotiation.cc:299] Negotiation successful
0619 12:32:55.909363 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:32:55.910355 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:55.898177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:36447)
0619 12:32:55.898821 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:55.898860 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:55.899599 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:55.900058 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:55.900065 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:55.900411 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:32:55.900900 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:55.900914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.901743 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.901746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:55.902634 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:55.902643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.903126 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.903133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:55.903337 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:55.904018 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:32:55.904034 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:32:55.905675 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:32:55.907448 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:55.907455 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:55.907471 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:55.907885 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:55.908206 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:32:55.908208 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:32:55.908210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:32:55.908392 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:32:55.908786 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:32:55.908791 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:32:55.909024 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 12:32:55.909219 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 12:32:55.909493 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":114,"threads_started":1}
I20260619 12:32:55.911492 16253 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36539
I20260619 12:32:55.911726 16253 heartbeater.cc:464] Registering TS with master...
I20260619 12:32:55.912187 16253 heartbeater.cc:511] Master 127.0.98.62:36539 requested a full tablet report, sending...
I20260619 12:32:55.913230 15794 ts_manager.cc:194] Registered new tserver with Master: 53c12ab3bbf34eda92d5f2477281e683 (127.0.98.3:44023)
I20260619 12:32:55.913801 15794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36447
I20260619 12:32:55.917865 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:32:55.928681 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:55.921168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40844 (local address 127.0.98.62:36539)
0619 12:32:55.921291 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:55.921296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:55.921486 (+ 190us) server_negotiation.cc:407] Connection header received
0619 12:32:55.921628 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:55.921632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:55.921677 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:55.921761 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:32:55.922484 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.922989 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.923821 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.924037 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.924152 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:55.925510 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:32:55.925513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:32:55.925516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:32:55.925544 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:32:55.926867 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:55.927323 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:55.927334 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:55.927336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:55.927406 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:32:55.927681 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:32:55.927684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:32:55.927686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:32:55.927902 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:32:55.928043 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:55.928272 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 12:32:55.928392 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 12:32:55.931779 15794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:32:55.934795 15794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:32:55.949414 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:55.943885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54728 (local address 127.0.98.1:38819)
0619 12:32:55.944674 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:55.944689 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:55.945002 (+ 313us) server_negotiation.cc:407] Connection header received
0619 12:32:55.945078 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:55.945114 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:55.945278 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:55.945420 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:32:55.945751 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.946468 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.947997 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.948669 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.948774 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:55.948913 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:55.949146 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 12:32:55.949244 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":129,"threads_started":1}
I20260619 12:32:55.949410 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:55.943689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38819 (local address 127.0.0.1:54728)
0619 12:32:55.944814 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:55.944828 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:55.944913 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:55.945546 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:55.945548 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:55.945561 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:32:55.945633 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:55.945637 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.946598 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.946604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:55.947868 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:55.947878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.948794 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.948798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:55.948905 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:55.948949 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 12:32:55.949063 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 12:32:55.949249 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"mutex_wait_us":121,"thread_start_us":85,"threads_started":1}
I20260619 12:32:55.949957 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:55.943158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44023 (local address 127.0.0.1:47980)
0619 12:32:55.944163 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:55.944211 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:55.944357 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:55.945132 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:55.945135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:55.945151 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:32:55.945243 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:55.945248 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.946640 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.946643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:55.948270 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:55.948282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.949526 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.949531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:55.949627 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:55.949655 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 12:32:55.949734 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 12:32:55.949824 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":239,"thread_start_us":84,"threads_started":1}
I20260619 12:32:55.950039 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:55.943270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.0.98.3:44023)
0619 12:32:55.944051 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:55.944064 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:55.944499 (+ 435us) server_negotiation.cc:407] Connection header received
0619 12:32:55.944586 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:55.944592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:55.944791 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:55.944940 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:32:55.945376 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.946459 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.948400 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.949230 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.949331 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:55.949458 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:55.949779 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 12:32:55.949872 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":101,"threads_started":1}
I20260619 12:32:55.950551 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:55.943934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47476 (local address 127.0.98.2:43239)
0619 12:32:55.944322 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:55.944334 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:55.944358 (+ 24us) server_negotiation.cc:407] Connection header received
0619 12:32:55.944523 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:55.944529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:55.944717 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:55.944882 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:32:55.945226 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.946460 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.948418 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.949163 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.949257 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:55.949382 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:55.950320 (+ 938us) server_negotiation.cc:299] Negotiation successful
0619 12:32:55.950417 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":84,"threads_started":1}
I20260619 12:32:55.952379 16187 tablet_service.cc:1467] Processing CreateTablet for tablet 42d11619c6c64f75bd59f2c7f1bb9fba (DEFAULT_TABLE table=test-table [id=4ab386c5d649446c9cdb5b6dc066bd81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:32:55.952381 15919 tablet_service.cc:1467] Processing CreateTablet for tablet 42d11619c6c64f75bd59f2c7f1bb9fba (DEFAULT_TABLE table=test-table [id=4ab386c5d649446c9cdb5b6dc066bd81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:32:55.953171 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:55.943243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43239 (local address 127.0.0.1:47476)
0619 12:32:55.944188 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:55.944229 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:55.944355 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:55.944864 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:55.944870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:55.944892 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:32:55.945029 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:55.945036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.946622 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.946625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:55.948270 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:55.948282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:55.949445 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:55.949449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:55.949871 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:55.949895 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 12:32:55.949969 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 12:32:55.952999 (+ 3030us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":149,"thread_start_us":115,"threads_started":1}
I20260619 12:32:55.953435 15919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d11619c6c64f75bd59f2c7f1bb9fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:32:55.953444 16187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d11619c6c64f75bd59f2c7f1bb9fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:32:55.955907 16053 tablet_service.cc:1467] Processing CreateTablet for tablet 42d11619c6c64f75bd59f2c7f1bb9fba (DEFAULT_TABLE table=test-table [id=4ab386c5d649446c9cdb5b6dc066bd81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:32:55.956970 16053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d11619c6c64f75bd59f2c7f1bb9fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:32:55.960939 16271 tablet_bootstrap.cc:492] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683: Bootstrap starting.
I20260619 12:32:55.963272 16271 tablet_bootstrap.cc:654] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683: Neither blocks nor log segments found. Creating new log.
I20260619 12:32:55.963819 16270 tablet_bootstrap.cc:492] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531: Bootstrap starting.
I20260619 12:32:55.964023 16272 tablet_bootstrap.cc:492] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049: Bootstrap starting.
I20260619 12:32:55.964581 16271 log.cc:824] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:55.966300 16270 tablet_bootstrap.cc:654] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531: Neither blocks nor log segments found. Creating new log.
I20260619 12:32:55.966398 16271 tablet_bootstrap.cc:492] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683: No bootstrap required, opened a new log
I20260619 12:32:55.966585 16271 ts_tablet_manager.cc:1397] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:32:55.966774 16272 tablet_bootstrap.cc:654] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049: Neither blocks nor log segments found. Creating new log.
I20260619 12:32:55.967576 16270 log.cc:824] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:55.967576 16272 log.cc:824] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049: Log is configured to *not* fsync() on all Append() calls
I20260619 12:32:55.969347 16270 tablet_bootstrap.cc:492] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531: No bootstrap required, opened a new log
I20260619 12:32:55.969528 16270 ts_tablet_manager.cc:1397] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:32:55.970026 16271 raft_consensus.cc:348] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:32:55.970273 16271 raft_consensus.cc:374] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:32:55.970312 16272 tablet_bootstrap.cc:492] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049: No bootstrap required, opened a new log
I20260619 12:32:55.970324 16271 raft_consensus.cc:729] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c12ab3bbf34eda92d5f2477281e683, State: Initialized, Role: FOLLOWER
I20260619 12:32:55.970474 16272 ts_tablet_manager.cc:1397] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 12:32:55.970774 16271 consensus_queue.cc:260] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:32:55.971393 16253 heartbeater.cc:503] Master 127.0.98.62:36539 was elected leader, sending a full tablet report...
I20260619 12:32:55.972862 16271 ts_tablet_manager.cc:1428] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 12:32:55.973345 16270 raft_consensus.cc:348] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:32:55.973613 16270 raft_consensus.cc:374] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:32:55.973685 16270 raft_consensus.cc:729] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d44216947d9046ca9641bb826cfc5531, State: Initialized, Role: FOLLOWER
I20260619 12:32:55.974123 16270 consensus_queue.cc:260] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:32:55.974192 16272 raft_consensus.cc:348] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:32:55.974471 16272 raft_consensus.cc:374] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:32:55.974529 16272 raft_consensus.cc:729] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3875c93147994ff1b3b64768c373c049, State: Initialized, Role: FOLLOWER
I20260619 12:32:55.974951 16272 consensus_queue.cc:260] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:32:55.975099 16270 ts_tablet_manager.cc:1428] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:32:55.975435 15985 heartbeater.cc:503] Master 127.0.98.62:36539 was elected leader, sending a full tablet report...
I20260619 12:32:55.975617 16119 heartbeater.cc:503] Master 127.0.98.62:36539 was elected leader, sending a full tablet report...
I20260619 12:32:55.976785 16272 ts_tablet_manager.cc:1428] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 12:32:56.044694 16277 raft_consensus.cc:482] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:32:56.044899 16277 raft_consensus.cc:504] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:32:56.046110 16277 leader_election.cc:288] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53c12ab3bbf34eda92d5f2477281e683 (127.0.98.3:44023), 3875c93147994ff1b3b64768c373c049 (127.0.98.2:43239)
I20260619 12:32:56.051244 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:56.046482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37903 (local address 127.0.98.2:43239)
0619 12:32:56.046614 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:56.046620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:56.046775 (+ 155us) server_negotiation.cc:407] Connection header received
0619 12:32:56.046951 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:56.046953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:56.046997 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:56.047113 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:32:56.047460 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:56.047958 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:56.049802 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:56.050462 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:56.050577 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:56.050639 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:56.051017 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 12:32:56.051104 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:32:56.051244 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:56.046481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36137 (local address 127.0.98.3:44023)
0619 12:32:56.046621 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:56.046627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:56.046815 (+ 188us) server_negotiation.cc:407] Connection header received
0619 12:32:56.046984 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:56.046987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:56.047040 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:56.047127 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:32:56.047469 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:56.047974 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:56.049803 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:56.050461 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:56.050577 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:56.050639 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:56.051017 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 12:32:56.051104 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 12:32:56.051282 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:56.046348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44023 (local address 127.0.98.1:36137)
0619 12:32:56.046719 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:56.046730 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:56.046865 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:56.047140 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:56.047142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:56.047157 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:32:56.047333 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:56.047363 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:56.048091 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:56.048097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:56.049613 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:56.049625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:56.050593 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:56.050597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:56.050784 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:56.050869 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 12:32:56.050948 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 12:32:56.051086 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":281,"thread_start_us":118,"threads_started":1}
I20260619 12:32:56.051285 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:56.046341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43239 (local address 127.0.98.1:37903)
0619 12:32:56.046684 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:56.046695 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 12:32:56.046831 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:32:56.047124 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:32:56.047127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:32:56.047143 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:32:56.047347 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:56.047353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:56.048119 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:56.048122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:56.049613 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:32:56.049625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:32:56.050593 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:32:56.050597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:32:56.050805 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:56.050869 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 12:32:56.050948 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 12:32:56.051087 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":241,"spinlock_wait_cycles":1408,"thread_start_us":92,"threads_started":1}
I20260619 12:32:56.052023 16207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d11619c6c64f75bd59f2c7f1bb9fba" candidate_uuid: "d44216947d9046ca9641bb826cfc5531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c12ab3bbf34eda92d5f2477281e683" is_pre_election: true
I20260619 12:32:56.052284 16073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d11619c6c64f75bd59f2c7f1bb9fba" candidate_uuid: "d44216947d9046ca9641bb826cfc5531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3875c93147994ff1b3b64768c373c049" is_pre_election: true
I20260619 12:32:56.052397 16207 raft_consensus.cc:2454] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d44216947d9046ca9641bb826cfc5531 in term 0.
I20260619 12:32:56.052613 16073 raft_consensus.cc:2454] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d44216947d9046ca9641bb826cfc5531 in term 0.
I20260619 12:32:56.052945 15872 leader_election.cc:302] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c12ab3bbf34eda92d5f2477281e683, d44216947d9046ca9641bb826cfc5531; no voters:
I20260619 12:32:56.053259 16277 raft_consensus.cc:2781] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:32:56.053359 16277 raft_consensus.cc:482] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:32:56.053404 16277 raft_consensus.cc:3037] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:32:56.054417 16277 raft_consensus.cc:504] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:32:56.054778 16277 leader_election.cc:288] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [CANDIDATE]: Term 1 election: Requested vote from peers 53c12ab3bbf34eda92d5f2477281e683 (127.0.98.3:44023), 3875c93147994ff1b3b64768c373c049 (127.0.98.2:43239)
I20260619 12:32:56.055156 16207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d11619c6c64f75bd59f2c7f1bb9fba" candidate_uuid: "d44216947d9046ca9641bb826cfc5531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c12ab3bbf34eda92d5f2477281e683"
I20260619 12:32:56.055262 16207 raft_consensus.cc:3037] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:32:56.056309 16207 raft_consensus.cc:2454] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d44216947d9046ca9641bb826cfc5531 in term 1.
I20260619 12:32:56.056631 15872 leader_election.cc:302] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c12ab3bbf34eda92d5f2477281e683, d44216947d9046ca9641bb826cfc5531; no voters:
I20260619 12:32:56.056816 16277 raft_consensus.cc:2781] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:32:56.056836 16073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d11619c6c64f75bd59f2c7f1bb9fba" candidate_uuid: "d44216947d9046ca9641bb826cfc5531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3875c93147994ff1b3b64768c373c049"
I20260619 12:32:56.057020 16073 raft_consensus.cc:3037] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:32:56.057070 16277 raft_consensus.cc:686] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [term 1 LEADER]: Becoming Leader. State: Replica: d44216947d9046ca9641bb826cfc5531, State: Running, Role: LEADER
I20260619 12:32:56.057451 16277 consensus_queue.cc:237] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:32:56.058254 16073 raft_consensus.cc:2454] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d44216947d9046ca9641bb826cfc5531 in term 1.
I20260619 12:32:56.060973 15794 catalog_manager.cc:5495] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 reported cstate change: term changed from 0 to 1, leader changed from <none> to d44216947d9046ca9641bb826cfc5531 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d44216947d9046ca9641bb826cfc5531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } health_report { overall_health: HEALTHY } } }
I20260619 12:32:56.101039 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:56.097331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54738 (local address 127.0.98.1:38819)
0619 12:32:56.097493 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 12:32:56.097499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:32:56.097521 (+ 22us) server_negotiation.cc:407] Connection header received
0619 12:32:56.097748 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:32:56.097751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:32:56.097810 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:32:56.097926 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:32:56.098169 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:56.098682 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:56.099586 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:32:56.099814 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:32:56.099925 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:32:56.100224 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:32:56.100649 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:32:56.100797 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 12:32:56.100834 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 12:32:56.100890 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 12:32:56.107031 16207 raft_consensus.cc:1261] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683 [term 1 FOLLOWER]: Refusing update from remote peer d44216947d9046ca9641bb826cfc5531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:32:56.107031 16073 raft_consensus.cc:1261] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049 [term 1 FOLLOWER]: Refusing update from remote peer d44216947d9046ca9641bb826cfc5531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:32:56.107786 16281 consensus_queue.cc:1035] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:32:56.108003 16277 consensus_queue.cc:1035] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:32:56.115046 16284 mvcc.cc:205] Tried to move back new op lower bound from 7298549252527173632 to 7298549252334665728. Current Snapshot: MvccSnapshot[applied={T|T < 7298549252527173632}]
I20260619 12:32:56.118151 16286 mvcc.cc:205] Tried to move back new op lower bound from 7298549252527173632 to 7298549252334665728. Current Snapshot: MvccSnapshot[applied={T|T < 7298549252527173632}]
I20260619 12:32:56.118337 16285 mvcc.cc:205] Tried to move back new op lower bound from 7298549252527173632 to 7298549252334665728. Current Snapshot: MvccSnapshot[applied={T|T < 7298549252527173632}]
I20260619 12:32:56.124039 15793 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:32:56.124220 15793 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:32:56.126660 15793 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: Sending DeleteTablet for 3 replicas of tablet 42d11619c6c64f75bd59f2c7f1bb9fba
I20260619 12:32:56.127399 16053 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d11619c6c64f75bd59f2c7f1bb9fba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:32:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47476
I20260619 12:32:56.127393 15919 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d11619c6c64f75bd59f2c7f1bb9fba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:32:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54728
I20260619 12:32:56.127409 16187 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d11619c6c64f75bd59f2c7f1bb9fba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:32:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47980
I20260619 12:32:56.127892 16295 tablet_replica.cc:330] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683: stopping tablet replica
I20260619 12:32:56.128201 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 15856
I20260619 12:32:56.128216 16295 raft_consensus.cc:2229] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:32:56.128255 16293 tablet_replica.cc:330] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531: stopping tablet replica
I20260619 12:32:56.128518 16295 raft_consensus.cc:2258] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:32:56.128538 16293 raft_consensus.cc:2229] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:32:56.128908 16294 tablet_replica.cc:330] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049: stopping tablet replica
I20260619 12:32:56.129243 16294 raft_consensus.cc:2229] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:32:56.129515 16294 raft_consensus.cc:2258] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:32:56.130090 16295 ts_tablet_manager.cc:1905] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:32:56.130952 16294 ts_tablet_manager.cc:1905] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:32:56.132212 16295 ts_tablet_manager.cc:1918] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:32:56.132318 16295 log.cc:1196] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/42d11619c6c64f75bd59f2c7f1bb9fba
I20260619 12:32:56.132602 16295 ts_tablet_manager.cc:1939] T 42d11619c6c64f75bd59f2c7f1bb9fba P 53c12ab3bbf34eda92d5f2477281e683: Deleting consensus metadata
I20260619 12:32:56.133721 15778 catalog_manager.cc:4841] TS 53c12ab3bbf34eda92d5f2477281e683 (127.0.98.3:44023): tablet 42d11619c6c64f75bd59f2c7f1bb9fba (table test-table [id=4ab386c5d649446c9cdb5b6dc066bd81]) successfully deleted
I20260619 12:32:56.134227 16294 ts_tablet_manager.cc:1918] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:32:56.134341 16294 log.cc:1196] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/42d11619c6c64f75bd59f2c7f1bb9fba
I20260619 12:32:56.134668 16294 ts_tablet_manager.cc:1939] T 42d11619c6c64f75bd59f2c7f1bb9fba P 3875c93147994ff1b3b64768c373c049: Deleting consensus metadata
I20260619 12:32:56.135982 15780 catalog_manager.cc:4841] TS 3875c93147994ff1b3b64768c373c049 (127.0.98.2:43239): tablet 42d11619c6c64f75bd59f2c7f1bb9fba (table test-table [id=4ab386c5d649446c9cdb5b6dc066bd81]) successfully deleted
W20260619 12:32:56.137416 15779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38819 (error 108)
I20260619 12:32:56.138219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 15990
I20260619 12:32:56.138468 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:32:56.138139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38819 (local address 127.0.0.1:54748)
0619 12:32:56.138284 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 12:32:56.138365 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 12:32:56.138958 15779 catalog_manager.cc:4584] TS d44216947d9046ca9641bb826cfc5531 (127.0.98.1:38819): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42d11619c6c64f75bd59f2c7f1bb9fba: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38819: connect: Connection refused (error 111)
I20260619 12:32:56.144599 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 16124
I20260619 12:32:56.150835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 15762
I20260619 12:33:16.158063 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36539
--webserver_interface=127.0.98.62
--webserver_port=46343
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:33:16.269340 16297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:16.269702 16297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:16.269785 16297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:16.274214 16297 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 12:33:16.274320 16297 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 12:33:16.274348 16297 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 12:33:16.274374 16297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:33:16.274420 16297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:16.274461 16297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:16.274500 16297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:33:16.274535 16297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:33:16.280164 16297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:16.281383 16297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:16.282444 16297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:16.296226 16304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:16.296252 16303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:16.296237 16306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:16.296531 16297 server_base.cc:1034] running on GCE node
I20260619 12:33:16.297451 16297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:16.298705 16297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:16.299952 16297 hybrid_clock.cc:648] HybridClock initialized: now 1781872396299932 us; error 37 us; skew 500 ppm
Jun 19 12:33:16 dist-test-slave-fg3h krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:33:16.307171 16297 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:33:16.308204 16297 webserver.cc:460] Webserver started at http://127.0.98.62:46343/ using document root <none> and password file <none>
I20260619 12:33:16.308851 16297 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:16.308974 16297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:16.313341 16297 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:33:16.315826 16313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:16.317361 16297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:33:16.317528 16297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "7c0fd18d86c046059f47a525549b2cc2"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "647da8ea1f0ff2bfac75336f6731794e"
server_key_iv: "c23d1abc81daf8647a863a8307d14bac"
server_key_version: "encryptionkey@0"
I20260619 12:33:16.318241 16297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:16.338791 16297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:16.342181 16297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:16.342403 16297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:16.350010 16297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36539
I20260619 12:33:16.350294 16365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36539 every 8 connection(s)
I20260619 12:33:16.351230 16297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:33:16.355418 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 16297
I20260619 12:33:16.355983 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38819
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36539
--webserver_port=42423
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:33:16.356305 16366 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:33:16.358763 16366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: Bootstrap starting.
I20260619 12:33:16.377859 16366 log.cc:824] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:16.387389 16366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:33:16.387838 16366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: Bootstrap complete.
I20260619 12:33:16.392426 16366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0fd18d86c046059f47a525549b2cc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260619 12:33:16.392848 16366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0fd18d86c046059f47a525549b2cc2, State: Initialized, Role: FOLLOWER
I20260619 12:33:16.393420 16366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0fd18d86c046059f47a525549b2cc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260619 12:33:16.393548 16366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:33:16.393642 16366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:33:16.393769 16366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:33:16.395998 16366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0fd18d86c046059f47a525549b2cc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260619 12:33:16.396365 16366 leader_election.cc:302] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c0fd18d86c046059f47a525549b2cc2; no voters:
I20260619 12:33:16.396728 16366 leader_election.cc:288] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 12:33:16.396819 16370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 12:33:16.397073 16370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 2 LEADER]: Becoming Leader. State: Replica: 7c0fd18d86c046059f47a525549b2cc2, State: Running, Role: LEADER
I20260619 12:33:16.397411 16370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0fd18d86c046059f47a525549b2cc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260619 12:33:16.398138 16366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:33:16.398417 16370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c0fd18d86c046059f47a525549b2cc2. Latest consensus state: current_term: 2 leader_uuid: "7c0fd18d86c046059f47a525549b2cc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0fd18d86c046059f47a525549b2cc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } } }
I20260619 12:33:16.398550 16370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [sys.catalog]: This master's current role is: LEADER
I20260619 12:33:16.399006 16371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7c0fd18d86c046059f47a525549b2cc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0fd18d86c046059f47a525549b2cc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } } }
I20260619 12:33:16.399191 16371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [sys.catalog]: This master's current role is: LEADER
I20260619 12:33:16.400287 16374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:33:16.406208 16374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:33:16.406980 16374 catalog_manager.cc:1259] Loaded cluster ID: 45e497b2a82341568485c075dd2a4d85
I20260619 12:33:16.407054 16374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:33:16.409785 16374 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:33:16.410547 16374 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: Loaded TSK: 0
I20260619 12:33:16.425453 16374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: Generated new TSK 1
I20260619 12:33:16.425762 16374 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 12:33:16.482880 16368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:16.483143 16368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:16.483223 16368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:16.487797 16368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:16.487862 16368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:16.487964 16368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:33:16.493428 16368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42423
--tserver_master_addrs=127.0.98.62:36539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:16.494436 16368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:16.495429 16368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:16.505162 16396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:16.505172 16394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:16.505182 16393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:16.505684 16368 server_base.cc:1034] running on GCE node
I20260619 12:33:16.506258 16368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:16.507053 16368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:16.508275 16368 hybrid_clock.cc:648] HybridClock initialized: now 1781872396508243 us; error 44 us; skew 500 ppm
Jun 19 12:33:16 dist-test-slave-fg3h krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:33:16.511665 16368 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:33:16.512713 16368 webserver.cc:460] Webserver started at http://127.0.98.1:42423/ using document root <none> and password file <none>
I20260619 12:33:16.513342 16368 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:16.513458 16368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:16.517467 16368 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 12:33:16.519865 16403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:16.521335 16368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:33:16.521472 16368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "d44216947d9046ca9641bb826cfc5531"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "c9cf69287ca50f05a0b1f519dc6e7ef0"
server_key_iv: "440ec8a5df881798ffe2a32b745a31c0"
server_key_version: "encryptionkey@0"
I20260619 12:33:16.522128 16368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:16.540585 16368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:16.544054 16368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:16.544263 16368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:16.544867 16368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:33:16.546104 16410 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 12:33:16.550252 16368 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 12:33:16.550318 16368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260619 12:33:16.550365 16368 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 12:33:16.552109 16368 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 12:33:16.552168 16368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 12:33:16.552424 16410 tablet_bootstrap.cc:492] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531: Bootstrap starting.
I20260619 12:33:16.563310 16368 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38819
I20260619 12:33:16.563385 16517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38819 every 8 connection(s)
I20260619 12:33:16.564531 16368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:33:16.572947 16410 log.cc:824] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:16.573704 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 16368
I20260619 12:33:16.574249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43239
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36539
--webserver_port=40815
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:33:16 dist-test-slave-fg3h krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:33:16.581749 16410 tablet_bootstrap.cc:492] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:33:16.582216 16410 tablet_bootstrap.cc:492] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531: Bootstrap complete.
I20260619 12:33:16.582387 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:16.566789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41033 (local address 127.0.98.62:36539)
0619 12:33:16.567202 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:16.567226 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:16.567532 (+ 306us) server_negotiation.cc:407] Connection header received
0619 12:33:16.568640 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:16.568652 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:16.568956 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:16.569343 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:16.570475 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:16.571477 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:16.572630 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:16.573180 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:16.573391 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:16.576378 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:16.576385 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:16.576396 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:33:16.576424 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:16.578545 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:16.579235 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:16.579241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:16.579247 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:16.579397 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:16.579839 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:16.579842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:16.579844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:16.580271 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:16.580395 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:16.581161 (+ 766us) server_negotiation.cc:299] Negotiation successful
0619 12:33:16.581464 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":126,"threads_started":1}
I20260619 12:33:16.582484 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:16.566798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:41033)
0619 12:33:16.567367 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:16.567409 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:16.568223 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:16.569391 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:16.569398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:16.569836 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:33:16.570292 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:16.570305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:16.571651 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:16.571654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:16.572486 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:16.572506 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:16.573349 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:16.573353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:16.573577 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:16.574376 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:33:16.574395 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:33:16.576173 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:33:16.578679 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:16.578688 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:16.578706 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:16.579098 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:16.579522 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:16.579527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:16.579532 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:16.579724 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:16.580422 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:33:16.580431 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:33:16.580771 (+ 340us) client_negotiation.cc:769] Sending connection context
0619 12:33:16.580994 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 12:33:16.581411 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":140,"threads_started":1}
I20260619 12:33:16.583576 16410 ts_tablet_manager.cc:1397] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531: Time spent bootstrapping tablet: real 0.031s user 0.025s sys 0.004s
I20260619 12:33:16.585108 16518 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36539
I20260619 12:33:16.585345 16518 heartbeater.cc:464] Registering TS with master...
I20260619 12:33:16.585896 16518 heartbeater.cc:511] Master 127.0.98.62:36539 requested a full tablet report, sending...
I20260619 12:33:16.586289 16410 raft_consensus.cc:348] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:16.586807 16410 raft_consensus.cc:729] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d44216947d9046ca9641bb826cfc5531, State: Initialized, Role: FOLLOWER
I20260619 12:33:16.587296 16410 consensus_queue.cc:260] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:16.588357 16410 ts_tablet_manager.cc:1428] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 12:33:16.588740 16330 ts_manager.cc:194] Registered new tserver with Master: d44216947d9046ca9641bb826cfc5531 (127.0.98.1:38819)
I20260619 12:33:16.592877 16330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41033
I20260619 12:33:16.596505 16518 heartbeater.cc:503] Master 127.0.98.62:36539 was elected leader, sending a full tablet report...
Jun 19 12:33:16 dist-test-slave-fg3h krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:33:16.603874 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:16.592210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38819 (local address 127.0.0.1:57744)
0619 12:33:16.592747 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:16.592794 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:16.592934 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:16.593508 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:16.593514 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:16.594011 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:33:16.594145 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:16.594153 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:16.595634 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:16.595638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:16.596591 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:16.596602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:16.597380 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:16.597398 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:16.597548 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:16.598142 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:33:16.598154 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:33:16.600035 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:33:16.602070 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:16.602080 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:16.602086 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:16.602319 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:16.602643 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:16.602647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:16.602649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:16.602736 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:16.603214 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:33:16.603220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:33:16.603349 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 12:33:16.603490 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 12:33:16.603691 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":115,"threads_started":1}
I20260619 12:33:16.603879 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:16.592253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.0.98.1:38819)
0619 12:33:16.593064 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:16.593078 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:16.593128 (+ 50us) server_negotiation.cc:407] Connection header received
0619 12:33:16.593200 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:16.593204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:16.593355 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:16.593531 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:16.594287 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:16.595503 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:16.596731 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:16.597267 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:16.597395 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:16.600176 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:16.600185 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:16.600197 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:33:16.600229 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:16.601948 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:16.602447 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:16.602453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:16.602457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:16.602540 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:16.602843 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:16.602847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:16.602849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:16.603091 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:16.603222 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:16.603569 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 12:33:16.603723 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":109,"threads_started":1}
I20260619 12:33:16.604876 16452 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d11619c6c64f75bd59f2c7f1bb9fba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:32:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57744
I20260619 12:33:16.605885 16527 tablet_replica.cc:330] stopping tablet replica
I20260619 12:33:16.606179 16527 raft_consensus.cc:2229] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:33:16.606475 16527 raft_consensus.cc:2258] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:33:16.607481 16527 ts_tablet_manager.cc:1905] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:33:16.611316 16527 ts_tablet_manager.cc:1918] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:33:16.611452 16527 log.cc:1196] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/wal/wals/42d11619c6c64f75bd59f2c7f1bb9fba
I20260619 12:33:16.611771 16527 ts_tablet_manager.cc:1939] T 42d11619c6c64f75bd59f2c7f1bb9fba P d44216947d9046ca9641bb826cfc5531: Deleting consensus metadata
I20260619 12:33:16.613381 16315 catalog_manager.cc:4841] TS d44216947d9046ca9641bb826cfc5531 (127.0.98.1:38819): tablet 42d11619c6c64f75bd59f2c7f1bb9fba (table test-table [id=4ab386c5d649446c9cdb5b6dc066bd81]) successfully deleted
W20260619 12:33:16.692452 16523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:16.692689 16523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:16.692731 16523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:16.696918 16523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:16.696972 16523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:16.697063 16523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:33:16.702518 16523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40815
--tserver_master_addrs=127.0.98.62:36539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:16.703486 16523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:16.704454 16523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:16.713987 16532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:16.714041 16535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:16.714013 16533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:16.714241 16523 server_base.cc:1034] running on GCE node
I20260619 12:33:16.714823 16523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:16.715474 16523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:16.716655 16523 hybrid_clock.cc:648] HybridClock initialized: now 1781872396716609 us; error 60 us; skew 500 ppm
Jun 19 12:33:16 dist-test-slave-fg3h krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:33:16.719859 16523 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:33:16.720762 16523 webserver.cc:460] Webserver started at http://127.0.98.2:40815/ using document root <none> and password file <none>
I20260619 12:33:16.721413 16523 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:16.721493 16523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:16.725493 16523 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 12:33:16.727731 16542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:16.729066 16523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:33:16.729373 16523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "3875c93147994ff1b3b64768c373c049"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "bc4023187a69a8312bcdc67466eebcda"
server_key_iv: "cf3f5f9aeb75590c7e1694142d79b056"
server_key_version: "encryptionkey@0"
I20260619 12:33:16.729924 16523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:16.742098 16523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:16.745592 16523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:16.745800 16523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:16.746428 16523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:33:16.747288 16523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:33:16.747360 16523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:16.747442 16523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:33:16.747478 16523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:16.757005 16523 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43239
I20260619 12:33:16.757021 16655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43239 every 8 connection(s)
I20260619 12:33:16.758286 16523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:33:16.761270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 16523
I20260619 12:33:16.761755 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44023
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36539
--webserver_port=40559
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 12:33:16 dist-test-slave-fg3h krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:33:16.776232 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:16.763365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48053 (local address 127.0.98.62:36539)
0619 12:33:16.763531 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:16.763536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:16.763558 (+ 22us) server_negotiation.cc:407] Connection header received
0619 12:33:16.764579 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:16.764582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:16.764634 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:16.764713 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:16.766474 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:16.767047 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:16.769020 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:16.769339 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:16.769431 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:16.772469 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:16.772472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:16.772474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:33:16.772503 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:16.774097 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:16.774644 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:16.774648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:16.774650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:16.774735 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:16.775087 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:16.775091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:16.775093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:16.775282 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:16.775392 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:16.775903 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 12:33:16.776053 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 12:33:16.777166 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:16.762187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:48053)
0619 12:33:16.762998 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:16.763043 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:16.764325 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:16.764941 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:16.764953 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:16.765530 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:33:16.766266 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:16.766291 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:16.767189 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:16.767196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:16.768868 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:16.768887 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:16.769521 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:16.769528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:16.769780 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:16.770463 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:33:16.770491 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:33:16.772270 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:33:16.774232 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:16.774238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:16.774256 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:16.774534 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:16.774834 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:16.774838 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:16.774840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:16.774990 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:16.775397 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:33:16.775406 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:33:16.775661 (+ 255us) client_negotiation.cc:769] Sending connection context
0619 12:33:16.775855 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 12:33:16.776197 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":147,"threads_started":1}
I20260619 12:33:16.778577 16656 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36539
I20260619 12:33:16.778848 16656 heartbeater.cc:464] Registering TS with master...
I20260619 12:33:16.779383 16656 heartbeater.cc:511] Master 127.0.98.62:36539 requested a full tablet report, sending...
I20260619 12:33:16.780514 16330 ts_manager.cc:194] Registered new tserver with Master: 3875c93147994ff1b3b64768c373c049 (127.0.98.2:43239)
I20260619 12:33:16.781348 16330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48053
W20260619 12:33:16.880824 16659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:16.881170 16659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:16.881222 16659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:16.885212 16659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:16.885272 16659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:16.885331 16659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:33:16.890663 16659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40559
--tserver_master_addrs=127.0.98.62:36539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:16.891690 16659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:16.892732 16659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:16.902588 16664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:16.902627 16665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:16.902593 16667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:16.902932 16659 server_base.cc:1034] running on GCE node
I20260619 12:33:16.903409 16659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:16.904048 16659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:16.905153 16659 hybrid_clock.cc:648] HybridClock initialized: now 1781872396905134 us; error 30 us; skew 500 ppm
Jun 19 12:33:16 dist-test-slave-fg3h krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:33:16.907979 16659 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:33:16.908794 16659 webserver.cc:460] Webserver started at http://127.0.98.3:40559/ using document root <none> and password file <none>
I20260619 12:33:16.909401 16659 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:16.909474 16659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:16.913214 16659 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 12:33:16.915216 16674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:16.916450 16659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:33:16.916541 16659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "53c12ab3bbf34eda92d5f2477281e683"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "525b5e1ee93fdf8d4b4bacc398f2fed3"
server_key_iv: "12f0dce3f1276957870c44e1adad1d1d"
server_key_version: "encryptionkey@0"
I20260619 12:33:16.917186 16659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:16.964174 16659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:16.967605 16659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:16.967826 16659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:16.968458 16659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:33:16.969331 16659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:33:16.969379 16659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:16.969444 16659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:33:16.969480 16659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:16.978616 16659 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44023
I20260619 12:33:16.978642 16787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44023 every 8 connection(s)
I20260619 12:33:16.979640 16659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:33:16.980130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 16659
I20260619 12:33:16.989610 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:16.980938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55828 (local address 127.0.98.62:36539)
0619 12:33:16.981070 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:16.981077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:16.981392 (+ 315us) server_negotiation.cc:407] Connection header received
0619 12:33:16.981604 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:16.981607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:16.981664 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:16.981756 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:16.982494 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:16.983073 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:16.983943 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:16.984182 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:16.984292 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:16.985775 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:16.985779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:16.985782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:33:16.985810 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:16.987468 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:16.988019 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:16.988022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:16.988024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:16.988114 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:16.988537 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:16.988540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:16.988543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:16.988781 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:16.989060 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:16.989234 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 12:33:16.989415 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 12:33:16 dist-test-slave-fg3h krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:33:16.994971 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:16.981606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35503 (local address 127.0.98.62:36539)
0619 12:33:16.981851 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:16.981856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:16.982995 (+ 1139us) server_negotiation.cc:407] Connection header received
0619 12:33:16.983892 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:16.983895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:16.983946 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:16.984054 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:16.985503 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:16.986058 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:16.987852 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:16.988194 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:16.988310 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:16.991524 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:16.991528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:16.991531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:33:16.991559 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:16.992864 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:16.993421 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:16.993424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:16.993425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:16.993509 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:16.993862 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:16.993865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:16.993866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:16.994093 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:16.994176 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:16.994697 (+ 521us) server_negotiation.cc:299] Negotiation successful
0619 12:33:16.994799 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":89,"threads_started":1}
I20260619 12:33:16.995721 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:16.982116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:35503)
0619 12:33:16.982832 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:16.982884 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:16.983689 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:16.984247 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:16.984254 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:16.984610 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:33:16.985338 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:16.985357 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:16.986225 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:16.986230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:16.987723 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:16.987732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:16.988376 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:16.988504 (+ 128us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:16.988746 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:16.989458 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:33:16.989481 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:33:16.991312 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:33:16.992985 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:16.992990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:16.993002 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:16.993326 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:16.993604 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:16.993606 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:16.993608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:16.993740 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:16.994185 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:33:16.994190 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:33:16.994429 (+ 239us) client_negotiation.cc:769] Sending connection context
0619 12:33:16.994596 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 12:33:16.994854 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":135,"threads_started":1}
I20260619 12:33:16.997058 16788 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36539
I20260619 12:33:16.997361 16788 heartbeater.cc:464] Registering TS with master...
I20260619 12:33:16.997861 16788 heartbeater.cc:511] Master 127.0.98.62:36539 requested a full tablet report, sending...
I20260619 12:33:16.998927 16330 ts_manager.cc:194] Registered new tserver with Master: 53c12ab3bbf34eda92d5f2477281e683 (127.0.98.3:44023)
I20260619 12:33:16.999516 16330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35503
I20260619 12:33:17.003695 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:33:17.007970 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:17.004375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.0.98.62:36539)
0619 12:33:17.004517 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:17.004520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:17.004718 (+ 198us) server_negotiation.cc:407] Connection header received
0619 12:33:17.004818 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:17.004820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:17.004938 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:17.005064 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:33:17.005360 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.005917 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.006770 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.007048 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.007145 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:17.007466 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:33:17.007611 (+ 145us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 12:33:17.007839 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55832: token expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 12:33:17.008077 16791 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55832: token expired
W20260619 12:33:17.008406 16257 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:36539)
I20260619 12:33:17.015746 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:17.008849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55848 (local address 127.0.98.62:36539)
0619 12:33:17.008963 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:17.008967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:17.009000 (+ 33us) server_negotiation.cc:407] Connection header received
0619 12:33:17.009030 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:17.009032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:17.009070 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:17.009232 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:17.009899 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.010393 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.011188 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.011388 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.011465 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:17.012762 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:17.012765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:17.012767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:33:17.012787 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:17.014357 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:17.014749 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:17.014752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:17.014753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:17.014840 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:17.015073 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:17.015075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:17.015076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:17.015247 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:17.015353 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:17.015528 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 12:33:17.015623 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 12:33:17.018347 16330 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:33:17.022164 16330 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:33:17.030503 16452 tablet_service.cc:1467] Processing CreateTablet for tablet a56172cf4c534663aa0e36f619079256 (DEFAULT_TABLE table=test-table [id=3b6f1e1c6fdb4347ade80c34f0d66493]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:33:17.031180 16452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a56172cf4c534663aa0e36f619079256. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:33:17.034560 16410 tablet_bootstrap.cc:492] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531: Bootstrap starting.
I20260619 12:33:17.035526 16410 tablet_bootstrap.cc:654] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531: Neither blocks nor log segments found. Creating new log.
I20260619 12:33:17.035859 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:17.029822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54556 (local address 127.0.98.3:44023)
0619 12:33:17.030191 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:17.030205 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:17.030229 (+ 24us) server_negotiation.cc:407] Connection header received
0619 12:33:17.030319 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:17.030325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:17.030589 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:17.031075 (+ 486us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:33:17.031532 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.032435 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.034360 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.035146 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.035289 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:17.035451 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:17.035619 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 12:33:17.035718 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":142,"threads_started":1}
I20260619 12:33:17.035861 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:17.029502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44023 (local address 127.0.0.1:54556)
0619 12:33:17.029610 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:17.029626 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:17.029712 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:17.030760 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:17.030763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:17.030782 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:33:17.030912 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:17.030919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.032569 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.032573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:17.034215 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:17.034225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.035290 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.035294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:17.035406 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:17.035479 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 12:33:17.035585 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 12:33:17.035700 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":123}
I20260619 12:33:17.036448 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:17.029907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43239 (local address 127.0.0.1:43614)
0619 12:33:17.030376 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:17.030390 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:17.030479 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:17.031234 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:17.031237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:17.031255 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:33:17.031353 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:17.031359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.032714 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.032717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:17.034156 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:17.034168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.035324 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.035328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:17.036144 (+ 816us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:17.036163 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 12:33:17.036249 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 12:33:17.036320 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":702,"thread_start_us":127,"threads_started":1}
I20260619 12:33:17.036537 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:17.030378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.0.98.2:43239)
0619 12:33:17.030738 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:17.030751 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:17.030784 (+ 33us) server_negotiation.cc:407] Connection header received
0619 12:33:17.030856 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:17.030861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:17.031073 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:17.031225 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:33:17.031501 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.032602 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.034317 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.035179 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.035481 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:17.035653 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:17.036285 (+ 632us) server_negotiation.cc:299] Negotiation successful
0619 12:33:17.036392 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":136,"threads_started":1}
I20260619 12:33:17.038074 16410 tablet_bootstrap.cc:492] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531: No bootstrap required, opened a new log
I20260619 12:33:17.038195 16410 ts_tablet_manager.cc:1397] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 12:33:17.038615 16410 raft_consensus.cc:348] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:17.038743 16410 raft_consensus.cc:374] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:33:17.038810 16410 raft_consensus.cc:729] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d44216947d9046ca9641bb826cfc5531, State: Initialized, Role: FOLLOWER
I20260619 12:33:17.038960 16410 consensus_queue.cc:260] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:17.039218 16410 ts_tablet_manager.cc:1428] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 12:33:17.039191 16722 tablet_service.cc:1467] Processing CreateTablet for tablet a56172cf4c534663aa0e36f619079256 (DEFAULT_TABLE table=test-table [id=3b6f1e1c6fdb4347ade80c34f0d66493]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:33:17.040366 16722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a56172cf4c534663aa0e36f619079256. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:33:17.041558 16590 tablet_service.cc:1467] Processing CreateTablet for tablet a56172cf4c534663aa0e36f619079256 (DEFAULT_TABLE table=test-table [id=3b6f1e1c6fdb4347ade80c34f0d66493]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:33:17.042625 16590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a56172cf4c534663aa0e36f619079256. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:33:17.047190 16801 tablet_bootstrap.cc:492] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: Bootstrap starting.
I20260619 12:33:17.049639 16801 tablet_bootstrap.cc:654] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: Neither blocks nor log segments found. Creating new log.
I20260619 12:33:17.049686 16802 tablet_bootstrap.cc:492] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049: Bootstrap starting.
I20260619 12:33:17.050899 16801 log.cc:824] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:17.052111 16802 tablet_bootstrap.cc:654] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049: Neither blocks nor log segments found. Creating new log.
I20260619 12:33:17.052906 16802 log.cc:824] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:17.052991 16801 tablet_bootstrap.cc:492] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: No bootstrap required, opened a new log
I20260619 12:33:17.053236 16801 ts_tablet_manager.cc:1397] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 12:33:17.054718 16802 tablet_bootstrap.cc:492] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049: No bootstrap required, opened a new log
I20260619 12:33:17.054926 16802 ts_tablet_manager.cc:1397] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:33:17.056316 16801 raft_consensus.cc:348] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:17.056628 16801 raft_consensus.cc:374] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:33:17.056699 16801 raft_consensus.cc:729] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c12ab3bbf34eda92d5f2477281e683, State: Initialized, Role: FOLLOWER
I20260619 12:33:17.057251 16801 consensus_queue.cc:260] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:17.057915 16788 heartbeater.cc:503] Master 127.0.98.62:36539 was elected leader, sending a full tablet report...
I20260619 12:33:17.058111 16802 raft_consensus.cc:348] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:17.058297 16801 ts_tablet_manager.cc:1428] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:33:17.058355 16802 raft_consensus.cc:374] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:33:17.058413 16802 raft_consensus.cc:729] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3875c93147994ff1b3b64768c373c049, State: Initialized, Role: FOLLOWER
I20260619 12:33:17.058810 16802 consensus_queue.cc:260] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:17.059434 16656 heartbeater.cc:503] Master 127.0.98.62:36539 was elected leader, sending a full tablet report...
I20260619 12:33:17.059815 16802 ts_tablet_manager.cc:1428] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 12:33:17.065824 16519 tablet.cc:2369] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:33:17.230686 16789 tablet.cc:2369] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:33:17.259624 16657 tablet.cc:2369] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:33:17.291653 16806 raft_consensus.cc:482] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:33:17.291908 16806 raft_consensus.cc:504] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:17.293134 16806 leader_election.cc:288] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53c12ab3bbf34eda92d5f2477281e683 (127.0.98.3:44023), d44216947d9046ca9641bb826cfc5531 (127.0.98.1:38819)
I20260619 12:33:17.297223 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:17.293614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57259 (local address 127.0.98.3:44023)
0619 12:33:17.293755 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:17.293759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:17.293913 (+ 154us) server_negotiation.cc:407] Connection header received
0619 12:33:17.294076 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:17.294081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:17.294129 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:17.294218 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:33:17.294515 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.295060 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.296200 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.296613 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.296735 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:17.296800 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:17.296976 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 12:33:17.297046 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:33:17.297272 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:17.293409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44023 (local address 127.0.98.2:57259)
0619 12:33:17.293828 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:17.293840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:17.293960 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:17.294230 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:17.294233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:17.294248 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:33:17.294402 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:17.294412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.295182 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.295186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:17.296073 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:17.296082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.296744 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.296747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:17.296843 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:17.296865 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:33:17.296960 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 12:33:17.297039 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":116,"spinlock_wait_cycles":11136,"thread_start_us":69,"threads_started":1}
I20260619 12:33:17.297277 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:17.293407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38819 (local address 127.0.98.2:49647)
0619 12:33:17.293754 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:17.293771 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:17.293941 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:17.294255 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:17.294257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:17.294295 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:33:17.294433 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:17.294438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.295171 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.295176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:17.296082 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:17.296091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.296795 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.296798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:17.296898 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:17.296920 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:33:17.296982 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 12:33:17.297069 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":22,"spinlock_wait_cycles":768,"thread_start_us":77,"threads_started":1}
I20260619 12:33:17.297262 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:17.293517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49647 (local address 127.0.98.1:38819)
0619 12:33:17.293755 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:17.293761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:17.293877 (+ 116us) server_negotiation.cc:407] Connection header received
0619 12:33:17.294055 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:17.294060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:17.294124 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:17.294245 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:33:17.294551 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.295052 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.296210 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.296669 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.296796 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:17.296982 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:17.297019 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 12:33:17.297095 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":74,"threads_started":1}
I20260619 12:33:17.297987 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56172cf4c534663aa0e36f619079256" candidate_uuid: "3875c93147994ff1b3b64768c373c049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c12ab3bbf34eda92d5f2477281e683" is_pre_election: true
I20260619 12:33:17.298285 16472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56172cf4c534663aa0e36f619079256" candidate_uuid: "3875c93147994ff1b3b64768c373c049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d44216947d9046ca9641bb826cfc5531" is_pre_election: true
I20260619 12:33:17.298378 16742 raft_consensus.cc:2454] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3875c93147994ff1b3b64768c373c049 in term 0.
I20260619 12:33:17.298617 16472 raft_consensus.cc:2454] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3875c93147994ff1b3b64768c373c049 in term 0.
I20260619 12:33:17.298904 16543 leader_election.cc:302] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3875c93147994ff1b3b64768c373c049, 53c12ab3bbf34eda92d5f2477281e683; no voters:
I20260619 12:33:17.299239 16806 raft_consensus.cc:2781] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:33:17.299357 16806 raft_consensus.cc:482] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:33:17.299418 16806 raft_consensus.cc:3037] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:33:17.300482 16806 raft_consensus.cc:504] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:17.300817 16806 leader_election.cc:288] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [CANDIDATE]: Term 1 election: Requested vote from peers 53c12ab3bbf34eda92d5f2477281e683 (127.0.98.3:44023), d44216947d9046ca9641bb826cfc5531 (127.0.98.1:38819)
I20260619 12:33:17.301249 16472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56172cf4c534663aa0e36f619079256" candidate_uuid: "3875c93147994ff1b3b64768c373c049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d44216947d9046ca9641bb826cfc5531"
I20260619 12:33:17.301414 16472 raft_consensus.cc:3037] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:33:17.302543 16472 raft_consensus.cc:2454] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3875c93147994ff1b3b64768c373c049 in term 1.
I20260619 12:33:17.302860 16742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56172cf4c534663aa0e36f619079256" candidate_uuid: "3875c93147994ff1b3b64768c373c049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c12ab3bbf34eda92d5f2477281e683"
I20260619 12:33:17.302919 16544 leader_election.cc:302] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3875c93147994ff1b3b64768c373c049, d44216947d9046ca9641bb826cfc5531; no voters:
I20260619 12:33:17.303004 16742 raft_consensus.cc:3037] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:33:17.303112 16806 raft_consensus.cc:2781] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:33:17.303380 16806 raft_consensus.cc:686] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 1 LEADER]: Becoming Leader. State: Replica: 3875c93147994ff1b3b64768c373c049, State: Running, Role: LEADER
I20260619 12:33:17.303714 16806 consensus_queue.cc:237] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:17.304188 16742 raft_consensus.cc:2454] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3875c93147994ff1b3b64768c373c049 in term 1.
I20260619 12:33:17.306684 16328 catalog_manager.cc:5495] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3875c93147994ff1b3b64768c373c049 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3875c93147994ff1b3b64768c373c049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } health_report { overall_health: UNKNOWN } } }
I20260619 12:33:17.340801 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:17.337517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43618 (local address 127.0.98.2:43239)
0619 12:33:17.337664 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:17.337668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:17.337683 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:33:17.337758 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:17.337760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:17.337864 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:17.337963 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:33:17.338164 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.338667 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.339447 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:17.339665 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:17.339755 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:17.340038 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:33:17.340446 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:33:17.340542 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:17.340635 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 12:33:17.340687 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 12:33:17.346958 16472 raft_consensus.cc:1261] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531 [term 1 FOLLOWER]: Refusing update from remote peer 3875c93147994ff1b3b64768c373c049: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:33:17.347014 16742 raft_consensus.cc:1261] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683 [term 1 FOLLOWER]: Refusing update from remote peer 3875c93147994ff1b3b64768c373c049: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:33:17.347744 16810 consensus_queue.cc:1035] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:33:17.347986 16806 consensus_queue.cc:1035] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:33:17.353212 16813 mvcc.cc:205] Tried to move back new op lower bound from 7298549339524886528 to 7298549339358883840. Current Snapshot: MvccSnapshot[applied={T|T < 7298549339524886528}]
I20260619 12:33:17.357167 16816 mvcc.cc:205] Tried to move back new op lower bound from 7298549339524886528 to 7298549339358883840. Current Snapshot: MvccSnapshot[applied={T|T < 7298549339524886528}]
I20260619 12:33:17.363955 16814 mvcc.cc:205] Tried to move back new op lower bound from 7298549339524886528 to 7298549339358883840. Current Snapshot: MvccSnapshot[applied={T|T < 7298549339524886528}]
I20260619 12:33:17.364401 16329 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:33:17.364594 16329 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:33:17.367390 16329 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: Sending DeleteTablet for 3 replicas of tablet a56172cf4c534663aa0e36f619079256
I20260619 12:33:17.368186 16722 tablet_service.cc:1514] Processing DeleteTablet for tablet a56172cf4c534663aa0e36f619079256 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:33:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54556
I20260619 12:33:17.368256 16452 tablet_service.cc:1514] Processing DeleteTablet for tablet a56172cf4c534663aa0e36f619079256 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:33:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57744
I20260619 12:33:17.369071 16590 tablet_service.cc:1514] Processing DeleteTablet for tablet a56172cf4c534663aa0e36f619079256 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:33:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43614
I20260619 12:33:17.369231 16822 tablet_replica.cc:330] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531: stopping tablet replica
I20260619 12:33:17.369361 16822 raft_consensus.cc:2229] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:33:17.369490 16822 raft_consensus.cc:2258] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:33:17.369684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 16368
I20260619 12:33:17.369769 16823 tablet_replica.cc:330] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: stopping tablet replica
I20260619 12:33:17.372409 16823 raft_consensus.cc:2229] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:33:17.372789 16823 raft_consensus.cc:2258] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:33:17.374405 16823 ts_tablet_manager.cc:1905] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 12:33:17.377067 16315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38819 (error 108)
I20260619 12:33:17.377204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 16523
I20260619 12:33:17.378238 16823 ts_tablet_manager.cc:1918] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:33:17.378360 16823 log.cc:1196] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/a56172cf4c534663aa0e36f619079256
I20260619 12:33:17.378754 16823 ts_tablet_manager.cc:1939] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: Deleting consensus metadata
I20260619 12:33:17.379055 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:17.378759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38819 (local address 127.0.0.1:57746)
0619 12:33:17.378876 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:17.378952 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 12:33:17.379333 16315 catalog_manager.cc:4584] TS d44216947d9046ca9641bb826cfc5531 (127.0.98.1:38819): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a56172cf4c534663aa0e36f619079256: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38819: connect: Connection refused (error 111)
I20260619 12:33:17.380128 16314 catalog_manager.cc:4841] TS 53c12ab3bbf34eda92d5f2477281e683 (127.0.98.3:44023): tablet a56172cf4c534663aa0e36f619079256 (table test-table [id=3b6f1e1c6fdb4347ade80c34f0d66493]) successfully deleted
I20260619 12:33:17.383401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 16659
I20260619 12:33:17.383596 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:17.383298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43239 (local address 127.0.0.1:43626)
0619 12:33:17.383429 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:17.383507 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 12:33:17.389907 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 16297
I20260619 12:33:37.396679 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36539
--webserver_interface=127.0.98.62
--webserver_port=46343
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:33:37.508131 16826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:37.508427 16826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:37.508469 16826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:37.512634 16826 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 12:33:37.512701 16826 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 12:33:37.512717 16826 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 12:33:37.512730 16826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:33:37.512748 16826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:37.512806 16826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:37.512840 16826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:33:37.512861 16826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:33:37.518304 16826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:37.519507 16826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:37.520578 16826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:37.529402 16831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:37.529417 16832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:37.529440 16834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:37.529770 16826 server_base.cc:1034] running on GCE node
I20260619 12:33:37.530335 16826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:37.531464 16826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:37.532691 16826 hybrid_clock.cc:648] HybridClock initialized: now 1781872417532669 us; error 42 us; skew 500 ppm
Jun 19 12:33:37 dist-test-slave-fg3h krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:33:37.535531 16826 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:33:37.536499 16826 webserver.cc:460] Webserver started at http://127.0.98.62:46343/ using document root <none> and password file <none>
I20260619 12:33:37.537158 16826 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:37.537248 16826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:37.541323 16826 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:33:37.543695 16841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:37.545038 16826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:33:37.545228 16826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "7c0fd18d86c046059f47a525549b2cc2"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "647da8ea1f0ff2bfac75336f6731794e"
server_key_iv: "c23d1abc81daf8647a863a8307d14bac"
server_key_version: "encryptionkey@0"
I20260619 12:33:37.545816 16826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:37.573035 16826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:37.576556 16826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:37.576788 16826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:37.584208 16826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36539
I20260619 12:33:37.584559 16893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36539 every 8 connection(s)
I20260619 12:33:37.585454 16826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:33:37.590090 16894 sys_catalog.cc:263] Verifying existing consensus state
I20260619 12:33:37.591876 16894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: Bootstrap starting.
I20260619 12:33:37.593693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 16826
I20260619 12:33:37.594127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38819
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36539
--webserver_port=42423
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:33:37.610069 16894 log.cc:824] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:37.622063 16894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:33:37.622484 16894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: Bootstrap complete.
I20260619 12:33:37.626622 16894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0fd18d86c046059f47a525549b2cc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260619 12:33:37.627024 16894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0fd18d86c046059f47a525549b2cc2, State: Initialized, Role: FOLLOWER
I20260619 12:33:37.627592 16894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0fd18d86c046059f47a525549b2cc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260619 12:33:37.627725 16894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:33:37.627817 16894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:33:37.627938 16894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 2 FOLLOWER]: Advancing to term 3
I20260619 12:33:37.630040 16894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0fd18d86c046059f47a525549b2cc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260619 12:33:37.630394 16894 leader_election.cc:302] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c0fd18d86c046059f47a525549b2cc2; no voters:
I20260619 12:33:37.630745 16894 leader_election.cc:288] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 12:33:37.630829 16898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 12:33:37.631062 16898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [term 3 LEADER]: Becoming Leader. State: Replica: 7c0fd18d86c046059f47a525549b2cc2, State: Running, Role: LEADER
I20260619 12:33:37.631434 16898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0fd18d86c046059f47a525549b2cc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260619 12:33:37.632059 16894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:33:37.632858 16899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7c0fd18d86c046059f47a525549b2cc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0fd18d86c046059f47a525549b2cc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } } }
I20260619 12:33:37.632984 16899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [sys.catalog]: This master's current role is: LEADER
I20260619 12:33:37.633312 16900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c0fd18d86c046059f47a525549b2cc2. Latest consensus state: current_term: 3 leader_uuid: "7c0fd18d86c046059f47a525549b2cc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0fd18d86c046059f47a525549b2cc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } } }
I20260619 12:33:37.633483 16900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2 [sys.catalog]: This master's current role is: LEADER
I20260619 12:33:37.633786 16907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:33:37.638903 16907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:33:37.639809 16907 catalog_manager.cc:1259] Loaded cluster ID: 45e497b2a82341568485c075dd2a4d85
I20260619 12:33:37.639886 16907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:33:37.642534 16907 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:33:37.643261 16907 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: Loaded TSK: 0
I20260619 12:33:37.643343 16907 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7c0fd18d86c046059f47a525549b2cc2: Loaded TSK: 1
F20260619 12:33:37.644063 16907 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcb2f2be9ad google::LogMessage::Fail() at ??:0
@ 0x7fcb2f2c188a google::LogMessage::SendToLog() at ??:0
@ 0x7fcb2f2be4de google::LogMessage::Flush() at ??:0
@ 0x7fcb2f2bfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcb31f8218d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fcb31f7af0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fcb39c237bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fcb39bfd99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fcb39bfe178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fcb39c3fdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fcb39c5e82b std::function<>::operator()() at ??:0
@ 0x7fcb39bfdb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fcb39bfef3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcb39bfb579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fcb39c3f640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fcb315103d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcb31510c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fcb315123f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fcb31501512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcb32b456db start_thread at ??:0
@ 0x7fcb2ddbc61f clone at ??:0
W20260619 12:33:37.732702 16896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:37.732921 16896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:37.732964 16896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:37.737491 16896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:37.737540 16896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:37.737639 16896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:33:37.743098 16896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42423
--tserver_master_addrs=127.0.98.62:36539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:37.744264 16896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:37.745375 16896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:37.755281 16944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:37.755321 16941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:37.755281 16942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:37.755606 16896 server_base.cc:1034] running on GCE node
I20260619 12:33:37.755991 16896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:37.756673 16896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:37.757872 16896 hybrid_clock.cc:648] HybridClock initialized: now 1781872417757839 us; error 47 us; skew 500 ppm
Jun 19 12:33:37 dist-test-slave-fg3h krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:33:37.760946 16896 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:33:37.761932 16896 webserver.cc:460] Webserver started at http://127.0.98.1:42423/ using document root <none> and password file <none>
I20260619 12:33:37.762509 16896 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:37.762611 16896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:37.766837 16896 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:33:37.769734 16951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:37.771216 16896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:33:37.771360 16896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "d44216947d9046ca9641bb826cfc5531"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "c9cf69287ca50f05a0b1f519dc6e7ef0"
server_key_iv: "440ec8a5df881798ffe2a32b745a31c0"
server_key_version: "encryptionkey@0"
I20260619 12:33:37.771944 16896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:37.802284 16896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:37.805789 16896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:37.806000 16896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:37.806617 16896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:33:37.807762 16958 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 12:33:37.813484 16896 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 12:33:37.813656 16896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 12:33:37.813750 16896 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 12:33:37.815524 16896 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 12:33:37.815621 16896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 12:33:37.815924 16958 tablet_bootstrap.cc:492] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531: Bootstrap starting.
I20260619 12:33:37.827171 16896 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38819
I20260619 12:33:37.827194 17065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38819 every 8 connection(s)
I20260619 12:33:37.828303 16896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:33:37.831337 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 16896
I20260619 12:33:37.831776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43239
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36539
--webserver_port=40815
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:33:37.839423 16958 log.cc:824] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:37.849354 16958 tablet_bootstrap.cc:492] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:33:37.849815 16958 tablet_bootstrap.cc:492] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531: Bootstrap complete.
I20260619 12:33:37.851351 16958 ts_tablet_manager.cc:1397] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531: Time spent bootstrapping tablet: real 0.036s user 0.027s sys 0.004s
I20260619 12:33:37.854696 16958 raft_consensus.cc:348] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:37.855393 16958 raft_consensus.cc:729] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d44216947d9046ca9641bb826cfc5531, State: Initialized, Role: FOLLOWER
I20260619 12:33:37.855953 16958 consensus_queue.cc:260] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:37.857059 16958 ts_tablet_manager.cc:1428] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260619 12:33:37.893282 17068 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:33:37.830374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:56893)
0619 12:33:37.830867 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:37.830921 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:37.833479 (+ 2558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:37.892336 (+ 58857us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":151,"threads_started":1}
W20260619 12:33:37.893851 17066 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36539 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36539: Client connection negotiation failed: client connection to 127.0.98.62:36539: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 12:33:37.894610 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:37.894370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:48047)
0619 12:33:37.894492 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:37.894555 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 12:33:37.895321 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:37.895141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:52203)
0619 12:33:37.895233 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:37.895274 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 12:33:37.895543 17066 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 12:33:37.955284 17069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:37.955644 17069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:37.955720 17069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:37.961640 17069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:37.961714 17069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:37.961783 17069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:33:37.967413 17069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40815
--tserver_master_addrs=127.0.98.62:36539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:37.968429 17069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:37.969543 17069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:37.979303 17079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:37.979476 17069 server_base.cc:1034] running on GCE node
W20260619 12:33:37.979343 17077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:37.979732 17076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:37.980203 17069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:37.980868 17069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:37.982031 17069 hybrid_clock.cc:648] HybridClock initialized: now 1781872417981993 us; error 50 us; skew 500 ppm
Jun 19 12:33:37 dist-test-slave-fg3h krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:33:37.984860 17069 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:33:37.985862 17069 webserver.cc:460] Webserver started at http://127.0.98.2:40815/ using document root <none> and password file <none>
I20260619 12:33:37.986438 17069 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:37.986501 17069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:37.990386 17069 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:33:37.992488 17086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:37.993778 17069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:33:37.993913 17069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "3875c93147994ff1b3b64768c373c049"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "bc4023187a69a8312bcdc67466eebcda"
server_key_iv: "cf3f5f9aeb75590c7e1694142d79b056"
server_key_version: "encryptionkey@0"
I20260619 12:33:37.994465 17069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:38.012545 17069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:38.016243 17069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:38.016474 17069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:38.017164 17069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:33:38.018280 17093 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 12:33:38.022502 17069 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 12:33:38.022599 17069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 12:33:38.022686 17069 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 12:33:38.024401 17069 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 12:33:38.024477 17069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 12:33:38.024812 17093 tablet_bootstrap.cc:492] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049: Bootstrap starting.
I20260619 12:33:38.035837 17069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43239
I20260619 12:33:38.036011 17200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43239 every 8 connection(s)
I20260619 12:33:38.036871 17069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:33:38.039605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 17069
I20260619 12:33:38.040200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44023
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36539
--webserver_port=40559
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:33:38.040539 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:38.039090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:36731)
0619 12:33:38.039610 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:38.039754 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":165,"threads_started":1}
W20260619 12:33:38.041520 17201 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36539 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36539: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
I20260619 12:33:38.042335 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:38.042040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:45073)
0619 12:33:38.042201 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:38.042263 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 12:33:38.043176 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:38.042844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:47967)
0619 12:33:38.043004 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:38.043074 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 12:33:38.043391 17201 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 12:33:38.045687 17093 log.cc:824] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:38.055704 17093 tablet_bootstrap.cc:492] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:33:38.056151 17093 tablet_bootstrap.cc:492] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049: Bootstrap complete.
I20260619 12:33:38.057664 17093 ts_tablet_manager.cc:1397] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049: Time spent bootstrapping tablet: real 0.033s user 0.018s sys 0.015s
I20260619 12:33:38.061529 17093 raft_consensus.cc:348] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:38.062089 17093 raft_consensus.cc:729] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3875c93147994ff1b3b64768c373c049, State: Initialized, Role: FOLLOWER
I20260619 12:33:38.062606 17093 consensus_queue.cc:260] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:38.063658 17093 ts_tablet_manager.cc:1428] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260619 12:33:38.063848 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:38.063565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:35603)
0619 12:33:38.063699 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:38.063763 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 12:33:38.064538 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:38.064344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:46727)
0619 12:33:38.064447 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:38.064490 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 12:33:38.159901 17204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:38.160192 17204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:38.160236 17204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:38.164463 17204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:38.164530 17204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:38.164639 17204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:33:38.170120 17204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40559
--tserver_master_addrs=127.0.98.62:36539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:38.171231 17204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:38.172314 17204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:38.182220 17214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:38.182263 17212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:38.182243 17211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:38.182541 17204 server_base.cc:1034] running on GCE node
I20260619 12:33:38.182991 17204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:38.183622 17204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:38.184792 17204 hybrid_clock.cc:648] HybridClock initialized: now 1781872418184773 us; error 31 us; skew 500 ppm
Jun 19 12:33:38 dist-test-slave-fg3h krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:33:38.187711 17204 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:33:38.188612 17204 webserver.cc:460] Webserver started at http://127.0.98.3:40559/ using document root <none> and password file <none>
I20260619 12:33:38.189210 17204 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:38.189276 17204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:38.193022 17204 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:33:38.195179 17221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:38.196435 17204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 12:33:38.196555 17204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "53c12ab3bbf34eda92d5f2477281e683"
format_stamp: "Formatted at 2026-06-19 12:32:55 on dist-test-slave-fg3h"
server_key: "525b5e1ee93fdf8d4b4bacc398f2fed3"
server_key_iv: "12f0dce3f1276957870c44e1adad1d1d"
server_key_version: "encryptionkey@0"
I20260619 12:33:38.197206 17204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:38.211414 17204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:38.214906 17204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:38.215137 17204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:38.215759 17204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:33:38.216603 17204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:33:38.216655 17204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:38.216727 17204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:33:38.216763 17204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:38.226284 17204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44023
I20260619 12:33:38.226404 17334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44023 every 8 connection(s)
I20260619 12:33:38.227350 17204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:33:38.228539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 17204
I20260619 12:33:38.231328 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:38.229852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:59687)
0619 12:33:38.230491 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:38.230590 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":115,"threads_started":1}
W20260619 12:33:38.231998 17335 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36539 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36539: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
I20260619 12:33:38.232751 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:38.232474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:36367)
0619 12:33:38.232640 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:38.232698 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 12:33:38.233527 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:38.233308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:33775)
0619 12:33:38.233411 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:38.233472 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 12:33:38.233728 17335 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 12:33:38.896631 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:38.896173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:38951)
0619 12:33:38.896444 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:38.896503 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
I20260619 12:33:39.065816 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:39.065321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:34057)
0619 12:33:39.065627 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:39.065695 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":80,"threads_started":1}
I20260619 12:33:39.234669 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:39.234252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:53815)
0619 12:33:39.234501 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:39.234565 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":68,"threads_started":1}
I20260619 12:33:39.260808 17342 raft_consensus.cc:482] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:33:39.261060 17342 raft_consensus.cc:504] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:39.262472 17342 leader_election.cc:288] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53c12ab3bbf34eda92d5f2477281e683 (127.0.98.3:44023), d44216947d9046ca9641bb826cfc5531 (127.0.98.1:38819)
Jun 19 12:33:39 dist-test-slave-fg3h krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 19 12:33:39 dist-test-slave-fg3h krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 12:33:39.276142 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:39.263090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36427 (local address 127.0.98.3:44023)
0619 12:33:39.263414 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:39.263434 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:39.263481 (+ 47us) server_negotiation.cc:407] Connection header received
0619 12:33:39.264489 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:39.264496 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:39.264856 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:39.265166 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:39.265705 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:39.266685 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:39.267900 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:39.268353 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:39.268520 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:39.271435 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:39.271440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:39.271453 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:33:39.271482 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:39.273338 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:39.273913 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:39.273919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:39.273924 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:39.274069 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:39.274484 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:39.274488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:39.274489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:39.274969 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:39.275109 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:39.275684 (+ 575us) server_negotiation.cc:299] Negotiation successful
0619 12:33:39.275931 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":103,"threads_started":1}
I20260619 12:33:39.276153 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:39.262936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52361 (local address 127.0.98.1:38819)
0619 12:33:39.263589 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:39.263596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:39.263624 (+ 28us) server_negotiation.cc:407] Connection header received
0619 12:33:39.263882 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:39.263887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:39.264102 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:39.264251 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:39.265734 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:39.266709 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:39.267878 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:39.268362 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:39.268569 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:39.271494 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:39.271499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:39.271511 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:33:39.271538 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:39.273711 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:39.274192 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:39.274198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:39.274203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:39.274357 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:39.274691 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:39.274696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:39.274699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:39.275094 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:39.275241 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:39.275691 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 12:33:39.275968 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":89,"threads_started":1}
I20260619 12:33:39.276171 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:39.262651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44023 (local address 127.0.98.2:36427)
0619 12:33:39.262952 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:39.262985 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:39.263545 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:39.265166 (+ 1621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:39.265168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:39.265314 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:33:39.265545 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:39.265561 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:39.266823 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:39.266828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:39.267736 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:39.267746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:39.268495 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:39.268499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:39.268738 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:39.269525 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:33:39.269537 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:33:39.271211 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:33:39.273468 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:39.273474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:39.273488 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:39.273774 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:39.274174 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:39.274178 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:39.274181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:39.274377 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:39.275080 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:33:39.275087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:33:39.275392 (+ 305us) client_negotiation.cc:769] Sending connection context
0619 12:33:39.275579 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 12:33:39.275942 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":134,"spinlock_wait_cycles":48768}
I20260619 12:33:39.276535 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:39.262755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38819 (local address 127.0.98.2:52361)
0619 12:33:39.263058 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:39.263073 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:39.263543 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:39.264380 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:39.264390 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:39.264884 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:33:39.265550 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:39.265563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:39.266871 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:39.266873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:39.267733 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:39.267741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:39.268527 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:39.268531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:39.268736 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:39.269459 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:33:39.269475 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:33:39.271326 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:33:39.273814 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:39.273819 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:39.273821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:39.274049 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:39.274473 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:39.274476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:39.274478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:39.274563 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:39.275217 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:33:39.275220 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:33:39.275385 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 12:33:39.275588 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 12:33:39.276039 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":215,"spinlock_wait_cycles":17792,"thread_start_us":91,"threads_started":1}
I20260619 12:33:39.278153 17020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56172cf4c534663aa0e36f619079256" candidate_uuid: "3875c93147994ff1b3b64768c373c049" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d44216947d9046ca9641bb826cfc5531" is_pre_election: true
I20260619 12:33:39.278491 17020 raft_consensus.cc:2454] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3875c93147994ff1b3b64768c373c049 in term 1.
W20260619 12:33:39.278975 17087 leader_election.cc:341] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 53c12ab3bbf34eda92d5f2477281e683 (127.0.98.3:44023): Not found: Tablet not found: a56172cf4c534663aa0e36f619079256
I20260619 12:33:39.279315 17088 leader_election.cc:302] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3875c93147994ff1b3b64768c373c049, d44216947d9046ca9641bb826cfc5531; no voters: 53c12ab3bbf34eda92d5f2477281e683
I20260619 12:33:39.279562 17342 raft_consensus.cc:2781] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 12:33:39.279654 17342 raft_consensus.cc:482] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:33:39.279707 17342 raft_consensus.cc:3037] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 1 FOLLOWER]: Advancing to term 2
I20260619 12:33:39.281900 17342 raft_consensus.cc:504] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:39.282299 17342 leader_election.cc:288] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [CANDIDATE]: Term 2 election: Requested vote from peers 53c12ab3bbf34eda92d5f2477281e683 (127.0.98.3:44023), d44216947d9046ca9641bb826cfc5531 (127.0.98.1:38819)
I20260619 12:33:39.282763 17020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56172cf4c534663aa0e36f619079256" candidate_uuid: "3875c93147994ff1b3b64768c373c049" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d44216947d9046ca9641bb826cfc5531"
I20260619 12:33:39.282910 17020 raft_consensus.cc:3037] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531 [term 1 FOLLOWER]: Advancing to term 2
W20260619 12:33:39.283181 17087 leader_election.cc:341] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 53c12ab3bbf34eda92d5f2477281e683 (127.0.98.3:44023): Not found: Tablet not found: a56172cf4c534663aa0e36f619079256
I20260619 12:33:39.285207 17020 raft_consensus.cc:2454] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3875c93147994ff1b3b64768c373c049 in term 2.
I20260619 12:33:39.285552 17088 leader_election.cc:302] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3875c93147994ff1b3b64768c373c049, d44216947d9046ca9641bb826cfc5531; no voters: 53c12ab3bbf34eda92d5f2477281e683
I20260619 12:33:39.285772 17342 raft_consensus.cc:2781] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 12:33:39.286015 17342 raft_consensus.cc:686] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [term 2 LEADER]: Becoming Leader. State: Replica: 3875c93147994ff1b3b64768c373c049, State: Running, Role: LEADER
I20260619 12:33:39.286369 17342 consensus_queue.cc:237] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:39.286739 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:39.286457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:39069)
0619 12:33:39.286634 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:39.286677 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 12:33:39.687846 17020 raft_consensus.cc:1261] T a56172cf4c534663aa0e36f619079256 P d44216947d9046ca9641bb826cfc5531 [term 2 FOLLOWER]: Refusing update from remote peer 3875c93147994ff1b3b64768c373c049: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260619 12:33:39.688679 17342 consensus_queue.cc:1035] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260619 12:33:39.690966 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:39.690569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:49495)
0619 12:33:39.690783 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:39.690842 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":65,"threads_started":1}
W20260619 12:33:39.693718 17087 consensus_peers.cc:487] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049 -> Peer 53c12ab3bbf34eda92d5f2477281e683 (127.0.98.3:44023): Couldn't send request to peer 53c12ab3bbf34eda92d5f2477281e683. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a56172cf4c534663aa0e36f619079256. This is attempt 1: this message will repeat every 5th retry.
I20260619 12:33:39.693744 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:39.693224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:35473)
0619 12:33:39.693356 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:39.693675 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 12:33:39.693817 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:39.693551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:44857)
0619 12:33:39.693681 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:39.693740 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 12:33:40.118566 17354 ts_tablet_manager.cc:927] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: Initiating tablet copy from peer 3875c93147994ff1b3b64768c373c049 (127.0.98.2:43239)
I20260619 12:33:40.119181 17354 tablet_copy_client.cc:316] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.2:43239
Jun 19 12:33:40 dist-test-slave-fg3h krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260619 12:33:40.134984 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:40.119809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43239 (local address 127.0.98.3:34745)
0619 12:33:40.120132 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:40.120178 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:40.120313 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:40.120747 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:40.120752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:40.121228 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:33:40.121348 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:40.121354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:40.122484 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:40.122487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:40.123163 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:40.123173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:40.123791 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:40.123794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:40.123883 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:40.129525 (+ 5642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:33:40.129539 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:33:40.131051 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:33:40.133257 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:40.133266 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:40.133271 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:40.133510 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:40.133810 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:40.133813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:40.133815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:40.133903 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:40.134424 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:33:40.134429 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:33:40.134543 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 12:33:40.134681 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 12:33:40.134829 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":115,"threads_started":1}
I20260619 12:33:40.135042 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:40.119973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34745 (local address 127.0.98.2:43239)
0619 12:33:40.120298 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:40.120311 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:40.120336 (+ 25us) server_negotiation.cc:407] Connection header received
0619 12:33:40.120385 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:40.120390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:40.120593 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:40.120720 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:40.121497 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:40.122358 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:40.123284 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:40.123674 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:40.123767 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:40.131221 (+ 7454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:40.131231 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:40.131244 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:33:40.131271 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:40.133147 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:40.133614 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:40.133618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:40.133620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:40.133703 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:40.134019 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:40.134022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:40.134023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:40.134310 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:40.134424 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:40.134773 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 12:33:40.134916 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":125,"threads_started":1}
I20260619 12:33:40.136231 17175 tablet_copy_service.cc:140] P 3875c93147994ff1b3b64768c373c049: Received BeginTabletCopySession request for tablet a56172cf4c534663aa0e36f619079256 from peer 53c12ab3bbf34eda92d5f2477281e683 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34745)
I20260619 12:33:40.136379 17175 tablet_copy_service.cc:161] P 3875c93147994ff1b3b64768c373c049: Beginning new tablet copy session on tablet a56172cf4c534663aa0e36f619079256 from peer 53c12ab3bbf34eda92d5f2477281e683 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34745: session id = 53c12ab3bbf34eda92d5f2477281e683-a56172cf4c534663aa0e36f619079256
I20260619 12:33:40.137532 17175 tablet_copy_source_session.cc:205] T a56172cf4c534663aa0e36f619079256 P 3875c93147994ff1b3b64768c373c049: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 12:33:40.141498 17354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a56172cf4c534663aa0e36f619079256. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:33:40.147800 17354 tablet_copy_client.cc:791] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: tablet copy: Starting download of 0 data blocks...
I20260619 12:33:40.147994 17354 tablet_copy_client.cc:655] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: tablet copy: Starting download of 1 WAL segments...
I20260619 12:33:40.149860 17354 tablet_copy_client.cc:531] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 12:33:40.151762 17354 tablet_bootstrap.cc:492] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: Bootstrap starting.
I20260619 12:33:40.156947 17354 log.cc:824] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:40.163683 17354 tablet_bootstrap.cc:492] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 12:33:40.163952 17354 tablet_bootstrap.cc:492] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: Bootstrap complete.
I20260619 12:33:40.164229 17354 ts_tablet_manager.cc:1397] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260619 12:33:40.166481 17354 raft_consensus.cc:348] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:40.166734 17354 raft_consensus.cc:729] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c12ab3bbf34eda92d5f2477281e683, State: Initialized, Role: FOLLOWER
I20260619 12:33:40.167094 17354 consensus_queue.cc:260] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c12ab3bbf34eda92d5f2477281e683" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "3875c93147994ff1b3b64768c373c049" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43239 } } peers { permanent_uuid: "d44216947d9046ca9641bb826cfc5531" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38819 } }
I20260619 12:33:40.167987 17354 ts_tablet_manager.cc:1428] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260619 12:33:40.168206 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:40.167935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:32937)
0619 12:33:40.168077 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:40.168136 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 12:33:40.168697 17175 tablet_copy_service.cc:342] P 3875c93147994ff1b3b64768c373c049: Request end of tablet copy session 53c12ab3bbf34eda92d5f2477281e683-a56172cf4c534663aa0e36f619079256 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34745
I20260619 12:33:40.168819 17175 tablet_copy_service.cc:433] P 3875c93147994ff1b3b64768c373c049: ending tablet copy session 53c12ab3bbf34eda92d5f2477281e683-a56172cf4c534663aa0e36f619079256 on tablet a56172cf4c534663aa0e36f619079256 with peer 53c12ab3bbf34eda92d5f2477281e683
I20260619 12:33:40.168879 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:40.168641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:39403)
0619 12:33:40.168750 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:40.168799 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 12:33:40.551371 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:40.550922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:37735)
0619 12:33:40.551187 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:40.551259 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260619 12:33:40.551420 17289 raft_consensus.cc:1203] T a56172cf4c534663aa0e36f619079256 P 53c12ab3bbf34eda92d5f2477281e683 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260619 12:33:40.552464 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:40.552217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:57503)
0619 12:33:40.552348 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:40.552403 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 12:33:40.694991 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:40.694565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:45081)
0619 12:33:40.694826 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:40.694887 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":89,"threads_started":1}
I20260619 12:33:41.552611 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:41.552196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:58065)
0619 12:33:41.552443 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:41.552503 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":76,"threads_started":1}
I20260619 12:33:41.553581 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:41.553221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:59691)
0619 12:33:41.553451 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:41.553501 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
I20260619 12:33:41.696177 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:41.695750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:41669)
0619 12:33:41.696021 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:41.696081 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
I20260619 12:33:42.553893 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:42.553446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:58483)
0619 12:33:42.553711 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:42.553770 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":74,"threads_started":1}
I20260619 12:33:42.554757 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:42.554316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:34989)
0619 12:33:42.554623 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:42.554677 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":64,"threads_started":1}
I20260619 12:33:42.697453 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:42.696998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:53681)
0619 12:33:42.697302 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:42.697359 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":151,"threads_started":1}
I20260619 12:33:43.555302 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:43.554868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:57417)
0619 12:33:43.555142 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:43.555204 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
I20260619 12:33:43.556190 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:43.555968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:45641)
0619 12:33:43.556085 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:43.556134 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 12:33:43.556222 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:43.555815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:39859)
0619 12:33:43.556063 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:43.556126 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":65,"threads_started":1}
I20260619 12:33:43.699100 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:43.698488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:47675)
0619 12:33:43.698874 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:43.698938 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":90,"threads_started":1}
I20260619 12:33:44.557499 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:44.557063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:54451)
0619 12:33:44.557342 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:44.557403 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":81,"threads_started":1}
I20260619 12:33:44.557600 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:44.557063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:35679)
0619 12:33:44.557455 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:44.557509 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":91,"threads_started":1}
I20260619 12:33:44.700357 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:44.699939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:59479)
0619 12:33:44.700202 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:44.700262 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":84,"threads_started":1}
I20260619 12:33:45.558898 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:45.558396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:51131)
0619 12:33:45.558748 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:45.558800 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":81,"threads_started":1}
I20260619 12:33:45.559089 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:45.558384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:49323)
0619 12:33:45.558865 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:45.558943 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":82,"threads_started":1}
I20260619 12:33:45.701736 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:45.701199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:52771)
0619 12:33:45.701558 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:45.701617 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":78,"threads_started":1}
I20260619 12:33:46.560233 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:46.559744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:47813)
0619 12:33:46.560051 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:46.560123 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":85,"threads_started":1}
I20260619 12:33:46.560307 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:46.559914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:55833)
0619 12:33:46.560163 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:46.560219 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
I20260619 12:33:46.702916 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:46.702523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:55589)
0619 12:33:46.702766 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:46.702822 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
I20260619 12:33:47.561638 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:47.561137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:45655)
0619 12:33:47.561451 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:47.561519 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":84,"threads_started":1}
I20260619 12:33:47.561743 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:47.561304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:39347)
0619 12:33:47.561563 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:47.561634 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":95,"threads_started":1}
I20260619 12:33:47.704056 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:47.703643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:33581)
0619 12:33:47.703888 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:47.703961 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260619 12:33:48.563041 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:48.562595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:59579)
0619 12:33:48.562855 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:48.562921 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":94,"threads_started":1}
I20260619 12:33:48.563097 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:48.562588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:54657)
0619 12:33:48.562943 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:48.563004 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":103,"threads_started":1}
I20260619 12:33:48.705405 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:48.704979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:33631)
0619 12:33:48.705256 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:48.705310 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":114,"threads_started":1}
I20260619 12:33:49.564368 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:49.563887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:51413)
0619 12:33:49.564187 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:49.564255 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":104,"threads_started":1}
I20260619 12:33:49.565934 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:49.565463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:52205)
0619 12:33:49.565777 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:49.565839 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":83,"threads_started":1}
I20260619 12:33:49.706655 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:49.706203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:42699)
0619 12:33:49.706493 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:49.706557 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":87,"threads_started":1}
I20260619 12:33:50.565768 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:50.565341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:56089)
0619 12:33:50.565601 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:50.565661 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":71,"threads_started":1}
I20260619 12:33:50.567085 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:50.566705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:37709)
0619 12:33:50.566955 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:50.567005 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
I20260619 12:33:50.707840 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:50.707412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:34147)
0619 12:33:50.707682 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:50.707744 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
I20260619 12:33:51.567049 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:51.566617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:50567)
0619 12:33:51.566889 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:51.566953 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":71,"threads_started":1}
I20260619 12:33:51.568105 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:51.567719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:41205)
0619 12:33:51.567966 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:51.568026 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":109,"threads_started":1}
I20260619 12:33:51.709303 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:51.708802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:37651)
0619 12:33:51.709130 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:51.709203 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":118,"threads_started":1}
I20260619 12:33:52.568427 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:52.567928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:41627)
0619 12:33:52.568205 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:52.568284 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":107,"threads_started":1}
I20260619 12:33:52.569264 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:52.568849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:56743)
0619 12:33:52.569076 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:52.569188 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
I20260619 12:33:52.710594 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:52.710156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:37551)
0619 12:33:52.710438 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:52.710499 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":96,"threads_started":1}
I20260619 12:33:53.235792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 16896
I20260619 12:33:53.241678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 17069
I20260619 12:33:53.247372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 17204
2026-06-19T12:33:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59433 ms)
[----------] 30 tests from SecurityITest (244922 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:33:53 dist-test-slave-fg3h krb5kdc[17404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:33:53 dist-test-slave-fg3h krb5kdc[17404](info): set up 2 sockets
Jun 19 12:33:53 dist-test-slave-fg3h krb5kdc[17404](info): commencing operation
krb5kdc: starting...
W20260619 12:33:54.301469 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:33:54 dist-test-slave-fg3h krb5kdc[17404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:33:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:33:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:33:54.428941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37163
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:33:54.538832 17417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:54.539160 17417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:54.539237 17417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:54.543340 17417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:33:54.543452 17417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:54.543489 17417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:54.543530 17417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:33:54.543562 17417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:33:54.548871 17417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:54.549989 17417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:54.551028 17417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:54.559664 17422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:54.559725 17425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:54.559670 17423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:54.560098 17417 server_base.cc:1034] running on GCE node
I20260619 12:33:54.560587 17417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:54.561705 17417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:54.562903 17417 hybrid_clock.cc:648] HybridClock initialized: now 1781872434562886 us; error 31 us; skew 500 ppm
Jun 19 12:33:54 dist-test-slave-fg3h krb5kdc[17404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:33:54.565727 17417 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:33:54.566612 17417 webserver.cc:460] Webserver started at http://127.0.98.62:42027/ using document root <none> and password file <none>
I20260619 12:33:54.567183 17417 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:54.567273 17417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:54.567502 17417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:33:54.569617 17417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "41d85fa74af8473dbc9efe9fd8bb3e23"
format_stamp: "Formatted at 2026-06-19 12:33:54 on dist-test-slave-fg3h"
server_key: "0d6c13d2612161fedf44adb3b8bebf47"
server_key_iv: "4eb98c945c8aa1efb42b712711b41ebc"
server_key_version: "encryptionkey@0"
I20260619 12:33:54.570233 17417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "41d85fa74af8473dbc9efe9fd8bb3e23"
format_stamp: "Formatted at 2026-06-19 12:33:54 on dist-test-slave-fg3h"
server_key: "0d6c13d2612161fedf44adb3b8bebf47"
server_key_iv: "4eb98c945c8aa1efb42b712711b41ebc"
server_key_version: "encryptionkey@0"
I20260619 12:33:54.573838 17417 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 12:33:54.576166 17432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:54.577566 17417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:33:54.577710 17417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "41d85fa74af8473dbc9efe9fd8bb3e23"
format_stamp: "Formatted at 2026-06-19 12:33:54 on dist-test-slave-fg3h"
server_key: "0d6c13d2612161fedf44adb3b8bebf47"
server_key_iv: "4eb98c945c8aa1efb42b712711b41ebc"
server_key_version: "encryptionkey@0"
I20260619 12:33:54.577823 17417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:54.590422 17417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:54.593425 17417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:54.593632 17417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:54.600756 17417 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37163
I20260619 12:33:54.600783 17484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37163 every 8 connection(s)
I20260619 12:33:54.601753 17417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:33:54.604369 17485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:33:54.605008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 17417
I20260619 12:33:54.605159 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:33:54.605480 392 external_mini_cluster.cc:1428] Setting key 274639f84b0b4bd4f56e87999294956d
I20260619 12:33:54.610384 17485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23: Bootstrap starting.
I20260619 12:33:54.612588 17485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23: Neither blocks nor log segments found. Creating new log.
I20260619 12:33:54.613376 17485 log.cc:824] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23: Log is configured to *not* fsync() on all Append() calls
Jun 19 12:33:54 dist-test-slave-fg3h krb5kdc[17404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:33:54.615558 17485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23: No bootstrap required, opened a new log
I20260619 12:33:54.619676 17485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d85fa74af8473dbc9efe9fd8bb3e23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37163 } }
I20260619 12:33:54.619900 17485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:33:54.619908 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:54.606871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55314 (local address 127.0.98.62:37163)
0619 12:33:54.607288 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:54.607302 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:54.607335 (+ 33us) server_negotiation.cc:407] Connection header received
0619 12:33:54.607975 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:54.607991 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:54.608322 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:54.608668 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:54.609479 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:54.610467 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:54.611348 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:54.611820 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:54.611996 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:54.614762 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:54.614767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:54.614779 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:33:54.614806 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:54.616853 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:54.617443 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:54.617447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:54.617452 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:54.617568 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:54.617905 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:54.617907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:54.617908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:54.618282 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:54.618453 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:54.618780 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 12:33:54.619121 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":131,"threads_started":1}
I20260619 12:33:54.619999 17485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d85fa74af8473dbc9efe9fd8bb3e23, State: Initialized, Role: FOLLOWER
I20260619 12:33:54.620517 17485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d85fa74af8473dbc9efe9fd8bb3e23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37163 } }
I20260619 12:33:54.620671 17485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:33:54.620756 17485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:33:54.620880 17485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:33:54.622047 17485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d85fa74af8473dbc9efe9fd8bb3e23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37163 } }
I20260619 12:33:54.622419 17485 leader_election.cc:302] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41d85fa74af8473dbc9efe9fd8bb3e23; no voters:
I20260619 12:33:54.622766 17485 leader_election.cc:288] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:33:54.622887 17490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:33:54.623147 17490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23 [term 1 LEADER]: Becoming Leader. State: Replica: 41d85fa74af8473dbc9efe9fd8bb3e23, State: Running, Role: LEADER
I20260619 12:33:54.623493 17490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d85fa74af8473dbc9efe9fd8bb3e23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37163 } }
I20260619 12:33:54.623975 17485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:33:54.625077 17491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41d85fa74af8473dbc9efe9fd8bb3e23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d85fa74af8473dbc9efe9fd8bb3e23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37163 } } }
I20260619 12:33:54.625254 17491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23 [sys.catalog]: This master's current role is: LEADER
I20260619 12:33:54.625579 17492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41d85fa74af8473dbc9efe9fd8bb3e23. Latest consensus state: current_term: 1 leader_uuid: "41d85fa74af8473dbc9efe9fd8bb3e23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d85fa74af8473dbc9efe9fd8bb3e23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37163 } } }
I20260619 12:33:54.625722 17492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23 [sys.catalog]: This master's current role is: LEADER
I20260619 12:33:54.625990 17499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:33:54.629148 17499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:33:54.634554 17499 catalog_manager.cc:1347] Generated new cluster ID: 0b44257b919e4723b5c9dadf321a9487
I20260619 12:33:54.634619 17499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:33:54.645330 17499 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:33:54.646649 17499 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:33:54.654021 17499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23: Generated new TSK 0
I20260619 12:33:54.654747 17499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:33:54.684870 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37163
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:33:54.798825 17511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:54.799101 17511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:54.799146 17511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:54.803505 17511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:54.803575 17511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:54.803675 17511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:33:54.809315 17511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:54.810526 17511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:54.811609 17511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:54.821553 17519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:54.821619 17517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:54.821686 17511 server_base.cc:1034] running on GCE node
W20260619 12:33:54.821866 17516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:54.822332 17511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:54.823050 17511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:54.824266 17511 hybrid_clock.cc:648] HybridClock initialized: now 1781872434824253 us; error 48 us; skew 500 ppm
Jun 19 12:33:54 dist-test-slave-fg3h krb5kdc[17404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:33:54.827376 17511 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:33:54.828317 17511 webserver.cc:460] Webserver started at http://127.0.98.1:40505/ using document root <none> and password file <none>
I20260619 12:33:54.828920 17511 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:54.829016 17511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:54.829281 17511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:33:54.831387 17511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "1ab50054214741018f42ccd5253da461"
format_stamp: "Formatted at 2026-06-19 12:33:54 on dist-test-slave-fg3h"
server_key: "4c3737dbc3b3e8df7e9d7b5a0456936f"
server_key_iv: "bd4077bbc591508bc04cc9d0413bc529"
server_key_version: "encryptionkey@0"
I20260619 12:33:54.831974 17511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1ab50054214741018f42ccd5253da461"
format_stamp: "Formatted at 2026-06-19 12:33:54 on dist-test-slave-fg3h"
server_key: "4c3737dbc3b3e8df7e9d7b5a0456936f"
server_key_iv: "bd4077bbc591508bc04cc9d0413bc529"
server_key_version: "encryptionkey@0"
I20260619 12:33:54.835973 17511 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:33:54.838649 17526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:54.839933 17511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:33:54.840078 17511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "1ab50054214741018f42ccd5253da461"
format_stamp: "Formatted at 2026-06-19 12:33:54 on dist-test-slave-fg3h"
server_key: "4c3737dbc3b3e8df7e9d7b5a0456936f"
server_key_iv: "bd4077bbc591508bc04cc9d0413bc529"
server_key_version: "encryptionkey@0"
I20260619 12:33:54.840201 17511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:54.864880 17511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:54.868484 17511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:54.868734 17511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:54.869417 17511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:33:54.870333 17511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:33:54.870406 17511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:54.870477 17511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:33:54.870529 17511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:54.879462 17511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44607
I20260619 12:33:54.879488 17639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44607 every 8 connection(s)
I20260619 12:33:54.880406 17511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:33:54.881631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 17511
I20260619 12:33:54.881765 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:33:54.882100 392 external_mini_cluster.cc:1428] Setting key 661d1df1e999c2f554b751702e7cb945
Jun 19 12:33:54 dist-test-slave-fg3h krb5kdc[17404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:33:54.896098 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:54.882591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43957 (local address 127.0.98.62:37163)
0619 12:33:54.882741 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:54.882746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:54.883803 (+ 1057us) server_negotiation.cc:407] Connection header received
0619 12:33:54.885872 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:54.885876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:54.885932 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:54.886010 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:54.887209 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:54.887756 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:54.888971 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:54.889280 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:54.889392 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:54.892017 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:54.892021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:54.892023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:33:54.892048 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:54.893676 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:54.894211 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:54.894214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:54.894215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:54.894343 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:54.894754 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:54.894757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:54.894759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:54.895021 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:54.895134 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:54.895681 (+ 547us) server_negotiation.cc:299] Negotiation successful
0619 12:33:54.895846 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 12:33:54.897114 17642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:54.883024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37163 (local address 127.0.98.1:43957)
0619 12:33:54.883626 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:54.883713 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:54.885632 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:54.886195 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:54.886202 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:54.886563 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:33:54.887048 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:54.887061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:54.887892 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:54.887895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:54.888819 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:54.888828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:54.889437 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:54.889441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:54.889628 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:54.890287 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:33:54.890303 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:33:54.891799 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:33:54.893794 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:54.893800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:54.893816 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:54.894084 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:54.894456 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:54.894463 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:54.894468 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:54.894656 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:54.895141 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:33:54.895148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:33:54.895399 (+ 251us) client_negotiation.cc:769] Sending connection context
0619 12:33:54.895566 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 12:33:54.895941 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":151,"threads_started":1}
I20260619 12:33:54.898434 17640 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37163
I20260619 12:33:54.898705 17640 heartbeater.cc:464] Registering TS with master...
I20260619 12:33:54.899194 17640 heartbeater.cc:511] Master 127.0.98.62:37163 requested a full tablet report, sending...
I20260619 12:33:54.900740 17449 ts_manager.cc:194] Registered new tserver with Master: 1ab50054214741018f42ccd5253da461 (127.0.98.1:44607)
I20260619 12:33:54.901964 17449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:33:54.912551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37163
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:33:55.044194 17645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:55.044631 17645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:55.044723 17645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:55.049221 17645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:55.049300 17645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:55.049417 17645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:33:55.055300 17645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:55.056494 17645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:55.057646 17645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:55.067497 17651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:55.067497 17653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:55.067524 17650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:55.067651 17645 server_base.cc:1034] running on GCE node
I20260619 12:33:55.068252 17645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:55.068990 17645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:55.070206 17645 hybrid_clock.cc:648] HybridClock initialized: now 1781872435070186 us; error 36 us; skew 500 ppm
Jun 19 12:33:55 dist-test-slave-fg3h krb5kdc[17404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:33:55.073298 17645 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:33:55.074206 17645 webserver.cc:460] Webserver started at http://127.0.98.2:43161/ using document root <none> and password file <none>
I20260619 12:33:55.074811 17645 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:55.074903 17645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:55.075132 17645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:33:55.077324 17645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "d44e6c58e9f04cdf8497d64032ed1d5d"
format_stamp: "Formatted at 2026-06-19 12:33:55 on dist-test-slave-fg3h"
server_key: "8f4a48b3f980b52a04a9f42a95ca73c4"
server_key_iv: "88a3136e67fa4099803e0fd34a7e13f5"
server_key_version: "encryptionkey@0"
I20260619 12:33:55.077890 17645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d44e6c58e9f04cdf8497d64032ed1d5d"
format_stamp: "Formatted at 2026-06-19 12:33:55 on dist-test-slave-fg3h"
server_key: "8f4a48b3f980b52a04a9f42a95ca73c4"
server_key_iv: "88a3136e67fa4099803e0fd34a7e13f5"
server_key_version: "encryptionkey@0"
I20260619 12:33:55.081789 17645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:33:55.084527 17660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:55.086015 17645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:33:55.086163 17645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "d44e6c58e9f04cdf8497d64032ed1d5d"
format_stamp: "Formatted at 2026-06-19 12:33:55 on dist-test-slave-fg3h"
server_key: "8f4a48b3f980b52a04a9f42a95ca73c4"
server_key_iv: "88a3136e67fa4099803e0fd34a7e13f5"
server_key_version: "encryptionkey@0"
I20260619 12:33:55.086287 17645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:55.108420 17645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:55.112008 17645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:55.112227 17645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:55.112856 17645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:33:55.113828 17645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:33:55.113904 17645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:55.113981 17645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:33:55.114025 17645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:55.123625 17645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42107
I20260619 12:33:55.123663 17773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42107 every 8 connection(s)
I20260619 12:33:55.124822 17645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:33:55.129987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 17645
I20260619 12:33:55.130110 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:33:55.130398 392 external_mini_cluster.cc:1428] Setting key a5606299d3aa9f002e83de00bfe059ee
Jun 19 12:33:55 dist-test-slave-fg3h krb5kdc[17404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:33:55.139889 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:55.126947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47939 (local address 127.0.98.62:37163)
0619 12:33:55.127104 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:55.127108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:55.128023 (+ 915us) server_negotiation.cc:407] Connection header received
0619 12:33:55.129042 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:55.129046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:55.129122 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:55.129204 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:55.130737 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.131394 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.132715 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.133131 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.133267 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:55.135754 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:55.135761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:55.135767 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:33:55.135801 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:55.137677 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:55.138209 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:55.138214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:55.138216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:55.138333 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:55.138677 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:55.138683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:55.138685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:55.138957 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:55.139059 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:55.139542 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 12:33:55.139693 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:33:55.140733 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:55.127348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37163 (local address 127.0.98.2:47939)
0619 12:33:55.127864 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:55.127912 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:55.128810 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:55.129384 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:55.129391 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:55.129840 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:33:55.130554 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:55.130574 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.131505 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.131509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:55.132578 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:55.132588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.133277 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.133281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:55.133467 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:55.134112 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:33:55.134128 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:33:55.135567 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:33:55.137809 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:55.137814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:55.137827 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:55.138101 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:55.138445 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:55.138447 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:55.138449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:55.138578 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:55.139066 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:33:55.139070 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:33:55.139300 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 12:33:55.139457 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 12:33:55.139709 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":125,"threads_started":1}
I20260619 12:33:55.142095 17774 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37163
I20260619 12:33:55.142364 17774 heartbeater.cc:464] Registering TS with master...
I20260619 12:33:55.142880 17774 heartbeater.cc:511] Master 127.0.98.62:37163 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 12:33:55.144151 17449 ts_manager.cc:194] Registered new tserver with Master: d44e6c58e9f04cdf8497d64032ed1d5d (127.0.98.2:42107)
I20260619 12:33:55.144963 17449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:33:55.159896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37163
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:33:55.270654 17779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:55.270993 17779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:55.271075 17779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:55.275323 17779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:55.275424 17779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:55.275521 17779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:33:55.281628 17779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:55.282773 17779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:55.283864 17779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:55.293690 17787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:55.293706 17784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:55.293892 17785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:55.293906 17779 server_base.cc:1034] running on GCE node
I20260619 12:33:55.294483 17779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:55.295107 17779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:55.296334 17779 hybrid_clock.cc:648] HybridClock initialized: now 1781872435296309 us; error 44 us; skew 500 ppm
Jun 19 12:33:55 dist-test-slave-fg3h krb5kdc[17404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:33:55.299435 17779 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:33:55.300354 17779 webserver.cc:460] Webserver started at http://127.0.98.3:39411/ using document root <none> and password file <none>
I20260619 12:33:55.300971 17779 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:55.301066 17779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:55.301311 17779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:33:55.303375 17779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "327a108e64b44c00b2198ca05d9c0e65"
format_stamp: "Formatted at 2026-06-19 12:33:55 on dist-test-slave-fg3h"
server_key: "74ef1124aba8ae52a44ce3d078e9bf33"
server_key_iv: "f3fff4488516cf551723bf337ec1cfd7"
server_key_version: "encryptionkey@0"
I20260619 12:33:55.303962 17779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "327a108e64b44c00b2198ca05d9c0e65"
format_stamp: "Formatted at 2026-06-19 12:33:55 on dist-test-slave-fg3h"
server_key: "74ef1124aba8ae52a44ce3d078e9bf33"
server_key_iv: "f3fff4488516cf551723bf337ec1cfd7"
server_key_version: "encryptionkey@0"
I20260619 12:33:55.307811 17779 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 12:33:55.310451 17794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:55.311684 17779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:33:55.311859 17779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "327a108e64b44c00b2198ca05d9c0e65"
format_stamp: "Formatted at 2026-06-19 12:33:55 on dist-test-slave-fg3h"
server_key: "74ef1124aba8ae52a44ce3d078e9bf33"
server_key_iv: "f3fff4488516cf551723bf337ec1cfd7"
server_key_version: "encryptionkey@0"
I20260619 12:33:55.311980 17779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:55.331312 17779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:55.334635 17779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:55.334842 17779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:55.335421 17779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:33:55.336265 17779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:33:55.336333 17779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:55.336404 17779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:33:55.336449 17779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:55.345177 17779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44277
I20260619 12:33:55.345232 17907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44277 every 8 connection(s)
I20260619 12:33:55.346135 17779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 12:33:55 dist-test-slave-fg3h krb5kdc[17404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:33:55.356094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 17779
I20260619 12:33:55.356233 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:33:55.356565 392 external_mini_cluster.cc:1428] Setting key 5ec53b0e818284788e66c9fa52c39519
I20260619 12:33:55.359953 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:55.348030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58861 (local address 127.0.98.62:37163)
0619 12:33:55.348161 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:55.348172 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:55.348960 (+ 788us) server_negotiation.cc:407] Connection header received
0619 12:33:55.350420 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:55.350424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:55.350485 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:55.350565 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:55.351732 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.352275 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.353506 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.353730 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.353844 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:55.356229 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:55.356232 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:55.356234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:33:55.356257 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:55.357962 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:55.358540 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:55.358543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:55.358544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:55.358643 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:55.359024 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:55.359026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:55.359027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:55.359199 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:55.359276 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:55.359741 (+ 465us) server_negotiation.cc:299] Negotiation successful
0619 12:33:55.359835 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:33:55.360785 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:55.348367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37163 (local address 127.0.98.3:58861)
0619 12:33:55.348801 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:55.348845 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:55.350198 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:55.350736 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:55.350744 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:55.351099 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:33:55.351580 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:55.351593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.352412 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.352415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:55.353377 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:55.353386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.353883 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.353887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:55.354065 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:55.354647 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:33:55.354664 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:33:55.356056 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:33:55.358079 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:55.358086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:55.358101 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:55.358447 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:55.358730 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:55.358735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:55.358737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:55.358939 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:55.359290 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:33:55.359295 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:33:55.359538 (+ 243us) client_negotiation.cc:769] Sending connection context
0619 12:33:55.359696 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 12:33:55.359966 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":118,"threads_started":1}
I20260619 12:33:55.361874 17908 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37163
I20260619 12:33:55.362120 17908 heartbeater.cc:464] Registering TS with master...
I20260619 12:33:55.362599 17908 heartbeater.cc:511] Master 127.0.98.62:37163 requested a full tablet report, sending...
I20260619 12:33:55.363577 17449 ts_manager.cc:194] Registered new tserver with Master: 327a108e64b44c00b2198ca05d9c0e65 (127.0.98.3:44277)
I20260619 12:33:55.364145 17449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58861
I20260619 12:33:55.370620 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:33:55.380119 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:55.372587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55330 (local address 127.0.98.62:37163)
0619 12:33:55.372730 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:55.372735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:55.372905 (+ 170us) server_negotiation.cc:407] Connection header received
0619 12:33:55.373078 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:55.373102 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:55.373153 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:55.373235 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:55.373912 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.374457 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.375298 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.375524 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.375632 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:55.377055 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:55.377058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:55.377061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:33:55.377143 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:55.378437 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:55.378855 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:55.378860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:55.378862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:55.378947 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:55.379221 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:55.379225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:55.379227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:55.379454 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:55.379593 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:55.379836 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 12:33:55.379952 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:33:55.383106 17449 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:33:55.386178 17449 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:33:55.401413 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:55.395669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44277 (local address 127.0.0.1:43594)
0619 12:33:55.395968 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:55.395980 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:55.396185 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:55.396906 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:55.396909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:55.396924 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:33:55.397032 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:55.397038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.398225 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.398231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:55.399816 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:55.399827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.400923 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.400927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:55.401003 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:55.401020 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 12:33:55.401105 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 12:33:55.401255 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":116,"thread_start_us":89,"threads_started":1}
I20260619 12:33:55.401477 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:55.396381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.0.98.1:44607)
0619 12:33:55.396766 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:55.396777 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:55.396796 (+ 19us) server_negotiation.cc:407] Connection header received
0619 12:33:55.396888 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:55.396893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:55.397073 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:55.397253 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:33:55.397481 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.398571 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.399829 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.400661 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.400778 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:55.400947 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:55.401227 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 12:33:55.401335 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":95,"threads_started":1}
I20260619 12:33:55.401613 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:55.395084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58462 (local address 127.0.98.2:42107)
0619 12:33:55.395857 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:55.395871 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:55.395898 (+ 27us) server_negotiation.cc:407] Connection header received
0619 12:33:55.395980 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:55.395987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:55.396365 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:55.396572 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:33:55.396875 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.398110 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.399944 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.400901 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.401048 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:55.401204 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:55.401354 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 12:33:55.401461 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":97,"threads_started":1}
I20260619 12:33:55.401831 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:55.395080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44607 (local address 127.0.0.1:39752)
0619 12:33:55.395451 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:55.395465 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:55.395572 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:55.397230 (+ 1658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:55.397233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:55.397251 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:33:55.397345 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:55.397351 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.398692 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.398695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:55.399700 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:55.399710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.400804 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.400808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:55.400945 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:55.400993 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 12:33:55.401141 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 12:33:55.401687 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":442,"thread_start_us":110,"threads_started":1}
I20260619 12:33:55.401988 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:55.396048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.0.98.3:44277)
0619 12:33:55.396475 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:55.396484 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:55.396529 (+ 45us) server_negotiation.cc:407] Connection header received
0619 12:33:55.396607 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:55.396612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:55.396791 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:55.396931 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:33:55.397165 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.398097 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.400136 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.400793 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.401500 (+ 707us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:55.401637 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:55.401691 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 12:33:55.401785 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":96,"threads_started":1}
I20260619 12:33:55.402879 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:55.394936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42107 (local address 127.0.0.1:58462)
0619 12:33:55.395400 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:55.395447 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:55.395569 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:55.396577 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:55.396583 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:55.396607 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:33:55.396743 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:55.396749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.398269 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.398272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:55.399816 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:55.399827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.401068 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.401072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:55.401197 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:55.401223 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 12:33:55.401326 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 12:33:55.401402 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":148,"thread_start_us":90,"threads_started":1}
I20260619 12:33:55.404800 17842 tablet_service.cc:1467] Processing CreateTablet for tablet ca13fb0ca33b477094e8627db0dfb30e (DEFAULT_TABLE table=test-table [id=cfbac6eee0f94a00aa515a5df98b1897]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:33:55.405212 17574 tablet_service.cc:1467] Processing CreateTablet for tablet ca13fb0ca33b477094e8627db0dfb30e (DEFAULT_TABLE table=test-table [id=cfbac6eee0f94a00aa515a5df98b1897]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:33:55.405268 17708 tablet_service.cc:1467] Processing CreateTablet for tablet ca13fb0ca33b477094e8627db0dfb30e (DEFAULT_TABLE table=test-table [id=cfbac6eee0f94a00aa515a5df98b1897]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:33:55.405944 17842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca13fb0ca33b477094e8627db0dfb30e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:33:55.406230 17574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca13fb0ca33b477094e8627db0dfb30e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:33:55.406230 17708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca13fb0ca33b477094e8627db0dfb30e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:33:55.412140 17925 tablet_bootstrap.cc:492] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65: Bootstrap starting.
I20260619 12:33:55.413630 17927 tablet_bootstrap.cc:492] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d: Bootstrap starting.
I20260619 12:33:55.413630 17926 tablet_bootstrap.cc:492] T ca13fb0ca33b477094e8627db0dfb30e P 1ab50054214741018f42ccd5253da461: Bootstrap starting.
I20260619 12:33:55.415145 17925 tablet_bootstrap.cc:654] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65: Neither blocks nor log segments found. Creating new log.
I20260619 12:33:55.415868 17927 tablet_bootstrap.cc:654] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d: Neither blocks nor log segments found. Creating new log.
I20260619 12:33:55.415864 17926 tablet_bootstrap.cc:654] T ca13fb0ca33b477094e8627db0dfb30e P 1ab50054214741018f42ccd5253da461: Neither blocks nor log segments found. Creating new log.
I20260619 12:33:55.416460 17925 log.cc:824] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:55.416759 17926 log.cc:824] T ca13fb0ca33b477094e8627db0dfb30e P 1ab50054214741018f42ccd5253da461: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:55.417050 17927 log.cc:824] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:55.418659 17925 tablet_bootstrap.cc:492] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65: No bootstrap required, opened a new log
I20260619 12:33:55.418861 17927 tablet_bootstrap.cc:492] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d: No bootstrap required, opened a new log
I20260619 12:33:55.418880 17925 ts_tablet_manager.cc:1397] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 12:33:55.419045 17927 ts_tablet_manager.cc:1397] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 12:33:55.420348 17926 tablet_bootstrap.cc:492] T ca13fb0ca33b477094e8627db0dfb30e P 1ab50054214741018f42ccd5253da461: No bootstrap required, opened a new log
I20260619 12:33:55.420531 17926 ts_tablet_manager.cc:1397] T ca13fb0ca33b477094e8627db0dfb30e P 1ab50054214741018f42ccd5253da461: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260619 12:33:55.422281 17927 raft_consensus.cc:348] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44e6c58e9f04cdf8497d64032ed1d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42107 } } peers { permanent_uuid: "1ab50054214741018f42ccd5253da461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "327a108e64b44c00b2198ca05d9c0e65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44277 } }
I20260619 12:33:55.422569 17927 raft_consensus.cc:374] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:33:55.422646 17927 raft_consensus.cc:729] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d44e6c58e9f04cdf8497d64032ed1d5d, State: Initialized, Role: FOLLOWER
I20260619 12:33:55.422984 17925 raft_consensus.cc:348] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44e6c58e9f04cdf8497d64032ed1d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42107 } } peers { permanent_uuid: "1ab50054214741018f42ccd5253da461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "327a108e64b44c00b2198ca05d9c0e65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44277 } }
I20260619 12:33:55.423126 17927 consensus_queue.cc:260] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44e6c58e9f04cdf8497d64032ed1d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42107 } } peers { permanent_uuid: "1ab50054214741018f42ccd5253da461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "327a108e64b44c00b2198ca05d9c0e65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44277 } }
I20260619 12:33:55.423247 17925 raft_consensus.cc:374] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:33:55.423352 17925 raft_consensus.cc:729] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327a108e64b44c00b2198ca05d9c0e65, State: Initialized, Role: FOLLOWER
I20260619 12:33:55.423811 17925 consensus_queue.cc:260] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44e6c58e9f04cdf8497d64032ed1d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42107 } } peers { permanent_uuid: "1ab50054214741018f42ccd5253da461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "327a108e64b44c00b2198ca05d9c0e65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44277 } }
I20260619 12:33:55.424238 17927 ts_tablet_manager.cc:1428] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 12:33:55.424507 17908 heartbeater.cc:503] Master 127.0.98.62:37163 was elected leader, sending a full tablet report...
I20260619 12:33:55.424602 17774 heartbeater.cc:503] Master 127.0.98.62:37163 was elected leader, sending a full tablet report...
I20260619 12:33:55.424975 17926 raft_consensus.cc:348] T ca13fb0ca33b477094e8627db0dfb30e P 1ab50054214741018f42ccd5253da461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44e6c58e9f04cdf8497d64032ed1d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42107 } } peers { permanent_uuid: "1ab50054214741018f42ccd5253da461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "327a108e64b44c00b2198ca05d9c0e65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44277 } }
I20260619 12:33:55.425238 17926 raft_consensus.cc:374] T ca13fb0ca33b477094e8627db0dfb30e P 1ab50054214741018f42ccd5253da461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:33:55.425287 17926 raft_consensus.cc:729] T ca13fb0ca33b477094e8627db0dfb30e P 1ab50054214741018f42ccd5253da461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ab50054214741018f42ccd5253da461, State: Initialized, Role: FOLLOWER
I20260619 12:33:55.425688 17926 consensus_queue.cc:260] T ca13fb0ca33b477094e8627db0dfb30e P 1ab50054214741018f42ccd5253da461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44e6c58e9f04cdf8497d64032ed1d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42107 } } peers { permanent_uuid: "1ab50054214741018f42ccd5253da461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "327a108e64b44c00b2198ca05d9c0e65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44277 } }
I20260619 12:33:55.426092 17925 ts_tablet_manager.cc:1428] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 12:33:55.426658 17926 ts_tablet_manager.cc:1428] T ca13fb0ca33b477094e8627db0dfb30e P 1ab50054214741018f42ccd5253da461: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260619 12:33:55.427009 17640 heartbeater.cc:503] Master 127.0.98.62:37163 was elected leader, sending a full tablet report...
W20260619 12:33:55.597164 17909 tablet.cc:2369] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:33:55.626232 17775 tablet.cc:2369] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:33:55.631814 17641 tablet.cc:2369] T ca13fb0ca33b477094e8627db0dfb30e P 1ab50054214741018f42ccd5253da461: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:33:55.696049 17931 raft_consensus.cc:482] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:33:55.696323 17931 raft_consensus.cc:504] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44e6c58e9f04cdf8497d64032ed1d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42107 } } peers { permanent_uuid: "1ab50054214741018f42ccd5253da461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "327a108e64b44c00b2198ca05d9c0e65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44277 } }
I20260619 12:33:55.697625 17931 leader_election.cc:288] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ab50054214741018f42ccd5253da461 (127.0.98.1:44607), 327a108e64b44c00b2198ca05d9c0e65 (127.0.98.3:44277)
I20260619 12:33:55.701381 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:55.697976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43597 (local address 127.0.98.1:44607)
0619 12:33:55.698103 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:55.698107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:55.698195 (+ 88us) server_negotiation.cc:407] Connection header received
0619 12:33:55.698339 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:55.698343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:55.698442 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:55.698540 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:33:55.698819 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.699326 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.700396 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.700818 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.700940 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:55.701016 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:55.701182 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 12:33:55.701252 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 12:33:55.701402 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:55.697841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44607 (local address 127.0.98.2:43597)
0619 12:33:55.698090 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:55.698107 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:55.698221 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:55.698555 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:55.698558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:55.698577 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:33:55.698704 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:55.698709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.699447 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.699452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:55.700260 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:55.700269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.700941 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.700945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:55.701043 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:55.701065 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:33:55.701168 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 12:33:55.701237 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260619 12:33:55.701864 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:55.698229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44277 (local address 127.0.98.2:48391)
0619 12:33:55.698527 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:55.698541 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:55.698636 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:55.698882 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:55.698885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:55.698900 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:33:55.699026 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:55.699032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.699734 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.699740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:55.700514 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:55.700522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.701301 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.701304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:55.701403 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:55.701426 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:33:55.701651 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 12:33:55.701717 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":79,"threads_started":1}
I20260619 12:33:55.701902 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:55.698420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48391 (local address 127.0.98.3:44277)
0619 12:33:55.698584 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:55.698590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:55.698607 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:33:55.698723 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:55.698726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:55.698776 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:55.698868 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:33:55.699141 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.699623 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.700643 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.701177 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.701292 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:55.701349 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:55.701705 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 12:33:55.701784 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 12:33:55.702008 17594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca13fb0ca33b477094e8627db0dfb30e" candidate_uuid: "d44e6c58e9f04cdf8497d64032ed1d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ab50054214741018f42ccd5253da461" is_pre_election: true
I20260619 12:33:55.702322 17594 raft_consensus.cc:2454] T ca13fb0ca33b477094e8627db0dfb30e P 1ab50054214741018f42ccd5253da461 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d44e6c58e9f04cdf8497d64032ed1d5d in term 0.
I20260619 12:33:55.702451 17862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca13fb0ca33b477094e8627db0dfb30e" candidate_uuid: "d44e6c58e9f04cdf8497d64032ed1d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327a108e64b44c00b2198ca05d9c0e65" is_pre_election: true
I20260619 12:33:55.702745 17862 raft_consensus.cc:2454] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d44e6c58e9f04cdf8497d64032ed1d5d in term 0.
I20260619 12:33:55.702795 17661 leader_election.cc:302] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ab50054214741018f42ccd5253da461, d44e6c58e9f04cdf8497d64032ed1d5d; no voters:
I20260619 12:33:55.703090 17931 raft_consensus.cc:2781] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:33:55.703171 17931 raft_consensus.cc:482] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:33:55.703202 17931 raft_consensus.cc:3037] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:33:55.704245 17931 raft_consensus.cc:504] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44e6c58e9f04cdf8497d64032ed1d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42107 } } peers { permanent_uuid: "1ab50054214741018f42ccd5253da461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "327a108e64b44c00b2198ca05d9c0e65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44277 } }
I20260619 12:33:55.704581 17931 leader_election.cc:288] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [CANDIDATE]: Term 1 election: Requested vote from peers 1ab50054214741018f42ccd5253da461 (127.0.98.1:44607), 327a108e64b44c00b2198ca05d9c0e65 (127.0.98.3:44277)
I20260619 12:33:55.704917 17594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca13fb0ca33b477094e8627db0dfb30e" candidate_uuid: "d44e6c58e9f04cdf8497d64032ed1d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ab50054214741018f42ccd5253da461"
I20260619 12:33:55.705020 17862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca13fb0ca33b477094e8627db0dfb30e" candidate_uuid: "d44e6c58e9f04cdf8497d64032ed1d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327a108e64b44c00b2198ca05d9c0e65"
I20260619 12:33:55.705075 17594 raft_consensus.cc:3037] T ca13fb0ca33b477094e8627db0dfb30e P 1ab50054214741018f42ccd5253da461 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:33:55.705173 17862 raft_consensus.cc:3037] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:33:55.706137 17594 raft_consensus.cc:2454] T ca13fb0ca33b477094e8627db0dfb30e P 1ab50054214741018f42ccd5253da461 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d44e6c58e9f04cdf8497d64032ed1d5d in term 1.
I20260619 12:33:55.706187 17862 raft_consensus.cc:2454] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d44e6c58e9f04cdf8497d64032ed1d5d in term 1.
I20260619 12:33:55.706454 17661 leader_election.cc:302] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 327a108e64b44c00b2198ca05d9c0e65, d44e6c58e9f04cdf8497d64032ed1d5d; no voters:
I20260619 12:33:55.706629 17931 raft_consensus.cc:2781] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:33:55.706837 17931 raft_consensus.cc:686] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [term 1 LEADER]: Becoming Leader. State: Replica: d44e6c58e9f04cdf8497d64032ed1d5d, State: Running, Role: LEADER
I20260619 12:33:55.707175 17931 consensus_queue.cc:237] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44e6c58e9f04cdf8497d64032ed1d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42107 } } peers { permanent_uuid: "1ab50054214741018f42ccd5253da461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "327a108e64b44c00b2198ca05d9c0e65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44277 } }
I20260619 12:33:55.710194 17447 catalog_manager.cc:5495] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d reported cstate change: term changed from 0 to 1, leader changed from <none> to d44e6c58e9f04cdf8497d64032ed1d5d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d44e6c58e9f04cdf8497d64032ed1d5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d44e6c58e9f04cdf8497d64032ed1d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ab50054214741018f42ccd5253da461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "327a108e64b44c00b2198ca05d9c0e65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44277 } health_report { overall_health: UNKNOWN } } }
I20260619 12:33:55.768620 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:55.764984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58464 (local address 127.0.98.2:42107)
0619 12:33:55.765154 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:55.765161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:55.765177 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:33:55.765277 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:55.765280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:55.765334 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:55.765479 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:33:55.765749 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.766257 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.767136 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:55.767356 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:55.767469 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:55.767784 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:33:55.768230 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:33:55.768370 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:55.768415 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 12:33:55.768473 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 12:33:55.774998 17862 raft_consensus.cc:1261] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65 [term 1 FOLLOWER]: Refusing update from remote peer d44e6c58e9f04cdf8497d64032ed1d5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:33:55.774998 17594 raft_consensus.cc:1261] T ca13fb0ca33b477094e8627db0dfb30e P 1ab50054214741018f42ccd5253da461 [term 1 FOLLOWER]: Refusing update from remote peer d44e6c58e9f04cdf8497d64032ed1d5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:33:55.775717 17931 consensus_queue.cc:1035] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab50054214741018f42ccd5253da461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:33:55.775918 17936 consensus_queue.cc:1035] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "327a108e64b44c00b2198ca05d9c0e65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:33:55.786000 17941 mvcc.cc:205] Tried to move back new op lower bound from 7298549496927002624 to 7298549496659042304. Current Snapshot: MvccSnapshot[applied={T|T < 7298549496927002624}]
I20260619 12:33:55.792253 17447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:33:55.792472 17447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:33:55.794880 17447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41d85fa74af8473dbc9efe9fd8bb3e23: Sending DeleteTablet for 3 replicas of tablet ca13fb0ca33b477094e8627db0dfb30e
I20260619 12:33:55.795653 17708 tablet_service.cc:1514] Processing DeleteTablet for tablet ca13fb0ca33b477094e8627db0dfb30e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:33:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58462
I20260619 12:33:55.795717 17574 tablet_service.cc:1514] Processing DeleteTablet for tablet ca13fb0ca33b477094e8627db0dfb30e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:33:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39752
I20260619 12:33:55.796160 17949 tablet_replica.cc:330] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d: stopping tablet replica
I20260619 12:33:55.796151 17842 tablet_service.cc:1514] Processing DeleteTablet for tablet ca13fb0ca33b477094e8627db0dfb30e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:33:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43594
I20260619 12:33:55.796502 17949 raft_consensus.cc:2229] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:33:55.796689 17950 tablet_replica.cc:330] T ca13fb0ca33b477094e8627db0dfb30e P 1ab50054214741018f42ccd5253da461: stopping tablet replica
I20260619 12:33:55.796867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 17511
I20260619 12:33:55.796923 17949 raft_consensus.cc:2258] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:33:55.799441 17949 ts_tablet_manager.cc:1905] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:33:55.803020 17949 ts_tablet_manager.cc:1918] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:33:55.803126 17949 log.cc:1196] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/ca13fb0ca33b477094e8627db0dfb30e
I20260619 12:33:55.803419 17951 tablet_replica.cc:330] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65: stopping tablet replica
I20260619 12:33:55.803468 17949 ts_tablet_manager.cc:1939] T ca13fb0ca33b477094e8627db0dfb30e P d44e6c58e9f04cdf8497d64032ed1d5d: Deleting consensus metadata
I20260619 12:33:55.803800 17951 raft_consensus.cc:2229] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:33:55.804152 17951 raft_consensus.cc:2258] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:33:55.805020 17435 catalog_manager.cc:4841] TS d44e6c58e9f04cdf8497d64032ed1d5d (127.0.98.2:42107): tablet ca13fb0ca33b477094e8627db0dfb30e (table test-table [id=cfbac6eee0f94a00aa515a5df98b1897]) successfully deleted
W20260619 12:33:55.805639 17433 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44607 (error 108)
I20260619 12:33:55.806090 17951 ts_tablet_manager.cc:1905] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:33:55.806638 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:55.806353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44607 (local address 127.0.0.1:39754)
0619 12:33:55.806485 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:55.806554 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 12:33:55.806854 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 17645
W20260619 12:33:55.808312 17433 catalog_manager.cc:4584] TS 1ab50054214741018f42ccd5253da461 (127.0.98.1:44607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca13fb0ca33b477094e8627db0dfb30e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44607: connect: Connection refused (error 111)
I20260619 12:33:55.808871 17951 ts_tablet_manager.cc:1918] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:33:55.808983 17951 log.cc:1196] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/ca13fb0ca33b477094e8627db0dfb30e
I20260619 12:33:55.809317 17951 ts_tablet_manager.cc:1939] T ca13fb0ca33b477094e8627db0dfb30e P 327a108e64b44c00b2198ca05d9c0e65: Deleting consensus metadata
I20260619 12:33:55.810379 17433 catalog_manager.cc:4841] TS 327a108e64b44c00b2198ca05d9c0e65 (127.0.98.3:44277): tablet ca13fb0ca33b477094e8627db0dfb30e (table test-table [id=cfbac6eee0f94a00aa515a5df98b1897]) successfully deleted
I20260619 12:33:55.813231 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 17779
I20260619 12:33:55.819254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 17417
2026-06-19T12:33:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2571 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T12:33:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:33:55Z Disabled control of system clock
I20260619 12:33:55.850020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41883
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 12:33:55.961282 17956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:55.961546 17956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:55.961621 17956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:55.965751 17956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:33:55.965807 17956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:55.965822 17956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:55.965839 17956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:33:55.965910 17956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:33:55.970635 17956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41883
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:55.971750 17956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:55.973201 17956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:55.981938 17961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:55.981938 17964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:55.981949 17962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:55.982131 17956 server_base.cc:1034] running on GCE node
I20260619 12:33:55.982730 17956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:55.983862 17956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:55.985062 17956 hybrid_clock.cc:648] HybridClock initialized: now 1781872435985034 us; error 56 us; skew 500 ppm
I20260619 12:33:55.986917 17956 webserver.cc:460] Webserver started at http://127.0.98.62:38985/ using document root <none> and password file <none>
I20260619 12:33:55.987506 17956 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:55.987599 17956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:55.987833 17956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:33:55.989980 17956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "a226584bc97541979f69aa8dde880822"
format_stamp: "Formatted at 2026-06-19 12:33:55 on dist-test-slave-fg3h"
server_key: "6823b0fa03df722295423e6f589786c6"
server_key_iv: "a2b934656a4f369c1b27b77123d4d0a7"
server_key_version: "encryptionkey@0"
I20260619 12:33:55.990542 17956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "a226584bc97541979f69aa8dde880822"
format_stamp: "Formatted at 2026-06-19 12:33:55 on dist-test-slave-fg3h"
server_key: "6823b0fa03df722295423e6f589786c6"
server_key_iv: "a2b934656a4f369c1b27b77123d4d0a7"
server_key_version: "encryptionkey@0"
I20260619 12:33:55.994180 17956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 12:33:55.996587 17970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:55.997932 17956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:33:55.998093 17956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "a226584bc97541979f69aa8dde880822"
format_stamp: "Formatted at 2026-06-19 12:33:55 on dist-test-slave-fg3h"
server_key: "6823b0fa03df722295423e6f589786c6"
server_key_iv: "a2b934656a4f369c1b27b77123d4d0a7"
server_key_version: "encryptionkey@0"
I20260619 12:33:55.998217 17956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:56.009529 17956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:56.010274 17956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:56.010651 17956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:56.018764 17956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41883
I20260619 12:33:56.018787 18022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41883 every 8 connection(s)
I20260619 12:33:56.019755 17956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:33:56.023034 18023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:33:56.026134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 17956
I20260619 12:33:56.026274 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:33:56.026561 392 external_mini_cluster.cc:1428] Setting key 42099ad029f55808bf68144572bdacec
I20260619 12:33:56.029562 18023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822: Bootstrap starting.
I20260619 12:33:56.031786 18023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822: Neither blocks nor log segments found. Creating new log.
I20260619 12:33:56.032539 18023 log.cc:824] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:56.034718 18023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822: No bootstrap required, opened a new log
I20260619 12:33:56.036518 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:56.028137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59738 (local address 127.0.98.62:41883)
0619 12:33:56.028891 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:56.028914 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:56.028952 (+ 38us) server_negotiation.cc:407] Connection header received
0619 12:33:56.029724 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:56.029744 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:56.030152 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:56.030548 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:56.030987 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.032029 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.032982 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.033550 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.033744 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:56.034686 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:56.034691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:56.034704 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:33:56.035169 (+ 465us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:56.035227 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:33:56.035248 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:56.035362 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:56.035526 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 12:33:56.035790 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":157,"threads_started":1}
I20260619 12:33:56.038542 18023 raft_consensus.cc:348] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a226584bc97541979f69aa8dde880822" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41883 } }
I20260619 12:33:56.038802 18023 raft_consensus.cc:374] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:33:56.038971 18023 raft_consensus.cc:729] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a226584bc97541979f69aa8dde880822, State: Initialized, Role: FOLLOWER
I20260619 12:33:56.039489 18023 consensus_queue.cc:260] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a226584bc97541979f69aa8dde880822" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41883 } }
I20260619 12:33:56.039629 18023 raft_consensus.cc:388] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:33:56.039671 18023 raft_consensus.cc:482] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:33:56.039746 18023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:33:56.040755 18023 raft_consensus.cc:504] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a226584bc97541979f69aa8dde880822" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41883 } }
I20260619 12:33:56.041126 18023 leader_election.cc:302] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a226584bc97541979f69aa8dde880822; no voters:
I20260619 12:33:56.041484 18023 leader_election.cc:288] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:33:56.041631 18028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:33:56.041894 18028 raft_consensus.cc:686] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822 [term 1 LEADER]: Becoming Leader. State: Replica: a226584bc97541979f69aa8dde880822, State: Running, Role: LEADER
I20260619 12:33:56.042232 18028 consensus_queue.cc:237] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a226584bc97541979f69aa8dde880822" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41883 } }
I20260619 12:33:56.042760 18023 sys_catalog.cc:564] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:33:56.044072 18029 sys_catalog.cc:455] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a226584bc97541979f69aa8dde880822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a226584bc97541979f69aa8dde880822" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41883 } } }
I20260619 12:33:56.044198 18029 sys_catalog.cc:458] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822 [sys.catalog]: This master's current role is: LEADER
I20260619 12:33:56.044513 18030 sys_catalog.cc:455] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a226584bc97541979f69aa8dde880822. Latest consensus state: current_term: 1 leader_uuid: "a226584bc97541979f69aa8dde880822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a226584bc97541979f69aa8dde880822" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41883 } } }
I20260619 12:33:56.044615 18030 sys_catalog.cc:458] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822 [sys.catalog]: This master's current role is: LEADER
I20260619 12:33:56.044632 18037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:33:56.047489 18037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:33:56.053304 18037 catalog_manager.cc:1347] Generated new cluster ID: 5232a2df26054a06a6a297bdea2a4c69
I20260619 12:33:56.053391 18037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:33:56.074608 18037 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:33:56.075618 18037 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:33:56.080034 18037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822: Generated new TSK 0
I20260619 12:33:56.080801 18037 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:33:56.090140 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41883
--builtin_ntp_servers=127.0.98.20:43693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 12:33:56.204154 18047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:56.204419 18047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:56.204459 18047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:56.208760 18047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:56.208825 18047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:56.208882 18047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:33:56.213740 18047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:56.214847 18047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:56.216346 18047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:56.226398 18053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:56.226558 18047 server_base.cc:1034] running on GCE node
W20260619 12:33:56.226423 18055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:56.226423 18052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:56.227144 18047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:56.227800 18047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:56.229003 18047 hybrid_clock.cc:648] HybridClock initialized: now 1781872436228956 us; error 77 us; skew 500 ppm
I20260619 12:33:56.230926 18047 webserver.cc:460] Webserver started at http://127.0.98.1:46861/ using document root <none> and password file <none>
I20260619 12:33:56.231520 18047 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:56.231616 18047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:56.231854 18047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:33:56.233990 18047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "8a4064b1f2b44501b0f450fae169be0f"
format_stamp: "Formatted at 2026-06-19 12:33:56 on dist-test-slave-fg3h"
server_key: "891161c6b3e56c5d63015bec9eca2dd2"
server_key_iv: "dac2afdc3fe30bda9092d1ec92b3be27"
server_key_version: "encryptionkey@0"
I20260619 12:33:56.234561 18047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8a4064b1f2b44501b0f450fae169be0f"
format_stamp: "Formatted at 2026-06-19 12:33:56 on dist-test-slave-fg3h"
server_key: "891161c6b3e56c5d63015bec9eca2dd2"
server_key_iv: "dac2afdc3fe30bda9092d1ec92b3be27"
server_key_version: "encryptionkey@0"
I20260619 12:33:56.238389 18047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:33:56.241035 18061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:56.242501 18047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:33:56.242630 18047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "8a4064b1f2b44501b0f450fae169be0f"
format_stamp: "Formatted at 2026-06-19 12:33:56 on dist-test-slave-fg3h"
server_key: "891161c6b3e56c5d63015bec9eca2dd2"
server_key_iv: "dac2afdc3fe30bda9092d1ec92b3be27"
server_key_version: "encryptionkey@0"
I20260619 12:33:56.242731 18047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:56.267294 18047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:56.268069 18047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:56.268290 18047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:56.268900 18047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:33:56.269874 18047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:33:56.269949 18047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:56.270018 18047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:33:56.270066 18047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:56.278879 18047 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40183
I20260619 12:33:56.278918 18174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40183 every 8 connection(s)
I20260619 12:33:56.279834 18047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:33:56.287062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 18047
I20260619 12:33:56.287170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:33:56.287472 392 external_mini_cluster.cc:1428] Setting key a33b4bec99cf4677492b71c6b4e007f8
I20260619 12:33:56.289610 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41883
--builtin_ntp_servers=127.0.98.20:43693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 12:33:56.289788 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:56.281716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52679 (local address 127.0.98.62:41883)
0619 12:33:56.281859 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:56.281863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:56.282973 (+ 1110us) server_negotiation.cc:407] Connection header received
0619 12:33:56.284473 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:56.284476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:56.284539 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:56.284652 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:56.285734 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.286275 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.287599 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.287840 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.287960 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:56.289214 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:56.289223 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:56.289225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:33:56.289236 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:56.289256 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:33:56.289265 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:56.289347 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:56.289515 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 12:33:56.289601 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 12:33:56.290737 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:56.282057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41883 (local address 127.0.98.1:52679)
0619 12:33:56.282804 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:56.282848 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:56.284229 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:56.284860 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:56.284867 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:56.284957 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:33:56.285573 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:56.285587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.286425 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.286428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:56.287447 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:56.287458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.288006 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.288010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:56.288174 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:56.288890 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:33:56.288892 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:33:56.288951 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:33:56.289019 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:33:56.289351 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:33:56.289362 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:33:56.289368 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 12:33:56.289515 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 12:33:56.289879 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":116,"threads_started":1}
I20260619 12:33:56.292276 18175 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41883
I20260619 12:33:56.292531 18175 heartbeater.cc:464] Registering TS with master...
I20260619 12:33:56.293056 18175 heartbeater.cc:511] Master 127.0.98.62:41883 requested a full tablet report, sending...
I20260619 12:33:56.294833 17987 ts_manager.cc:194] Registered new tserver with Master: 8a4064b1f2b44501b0f450fae169be0f (127.0.98.1:40183)
I20260619 12:33:56.296627 17987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:52679
W20260619 12:33:56.404780 18178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:56.405047 18178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:56.405138 18178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:56.409224 18178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:56.409281 18178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:56.409336 18178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:33:56.414242 18178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:56.415364 18178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:56.416739 18178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:56.426514 18184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:56.426455 18186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:56.426451 18183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:56.426959 18178 server_base.cc:1034] running on GCE node
I20260619 12:33:56.427366 18178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:56.428010 18178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:56.429165 18178 hybrid_clock.cc:648] HybridClock initialized: now 1781872436429125 us; error 57 us; skew 500 ppm
I20260619 12:33:56.431085 18178 webserver.cc:460] Webserver started at http://127.0.98.2:35319/ using document root <none> and password file <none>
I20260619 12:33:56.431726 18178 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:56.431789 18178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:56.431958 18178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:33:56.434073 18178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "72f07b99c28b48aa849db764f420db7e"
format_stamp: "Formatted at 2026-06-19 12:33:56 on dist-test-slave-fg3h"
server_key: "eb7e93e6386c7c01bc4119798fbb20ba"
server_key_iv: "4660ef2fbf648d1698c25db16615344b"
server_key_version: "encryptionkey@0"
I20260619 12:33:56.434593 18178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "72f07b99c28b48aa849db764f420db7e"
format_stamp: "Formatted at 2026-06-19 12:33:56 on dist-test-slave-fg3h"
server_key: "eb7e93e6386c7c01bc4119798fbb20ba"
server_key_iv: "4660ef2fbf648d1698c25db16615344b"
server_key_version: "encryptionkey@0"
I20260619 12:33:56.438371 18178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 12:33:56.440824 18192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:56.442248 18178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:33:56.442415 18178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "72f07b99c28b48aa849db764f420db7e"
format_stamp: "Formatted at 2026-06-19 12:33:56 on dist-test-slave-fg3h"
server_key: "eb7e93e6386c7c01bc4119798fbb20ba"
server_key_iv: "4660ef2fbf648d1698c25db16615344b"
server_key_version: "encryptionkey@0"
I20260619 12:33:56.442546 18178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:56.456523 18178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:56.457341 18178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:56.457551 18178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:56.458168 18178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:33:56.459151 18178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:33:56.459225 18178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:56.459300 18178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:33:56.459344 18178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:56.468792 18178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45001
I20260619 12:33:56.468822 18305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45001 every 8 connection(s)
I20260619 12:33:56.469748 18178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:33:56.473364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 18178
I20260619 12:33:56.473476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:33:56.473822 392 external_mini_cluster.cc:1428] Setting key c154b9cc1246562b966b3353a5910a90
I20260619 12:33:56.475925 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41883
--builtin_ntp_servers=127.0.98.20:43693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 12:33:56.479833 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:56.471717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52605 (local address 127.0.98.62:41883)
0619 12:33:56.471895 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:56.471900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:56.473051 (+ 1151us) server_negotiation.cc:407] Connection header received
0619 12:33:56.474679 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:56.474683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:56.474729 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:56.474818 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:56.475728 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.476357 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.477618 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.477844 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.477958 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:56.479160 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:56.479163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:56.479165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:33:56.479177 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:56.479197 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:33:56.479205 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:56.479290 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:56.479527 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 12:33:56.479611 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 12:33:56.480688 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:56.472153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41883 (local address 127.0.98.2:52605)
0619 12:33:56.472881 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:56.472924 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:56.474488 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:56.474975 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:56.474982 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:56.475072 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:33:56.475577 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:56.475588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.476486 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.476492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:56.477488 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:56.477497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.478003 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.478007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:56.478178 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:56.478851 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:33:56.478853 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:33:56.478897 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:33:56.478968 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:33:56.479293 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:33:56.479305 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:33:56.479314 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 12:33:56.479485 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 12:33:56.479743 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":119,"threads_started":1}
I20260619 12:33:56.482028 18306 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41883
I20260619 12:33:56.482262 18306 heartbeater.cc:464] Registering TS with master...
I20260619 12:33:56.482808 18306 heartbeater.cc:511] Master 127.0.98.62:41883 requested a full tablet report, sending...
I20260619 12:33:56.483918 17987 ts_manager.cc:194] Registered new tserver with Master: 72f07b99c28b48aa849db764f420db7e (127.0.98.2:45001)
I20260619 12:33:56.484572 17987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:52605
W20260619 12:33:56.587890 18309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:56.588229 18309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:56.588313 18309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:56.592419 18309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:56.592501 18309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:56.592594 18309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:33:56.597316 18309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:56.598431 18309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:56.599975 18309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:56.609480 18314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:56.609632 18309 server_base.cc:1034] running on GCE node
W20260619 12:33:56.609470 18315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:56.609807 18317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:56.610241 18309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:56.610870 18309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:56.612030 18309 hybrid_clock.cc:648] HybridClock initialized: now 1781872436611995 us; error 54 us; skew 500 ppm
I20260619 12:33:56.613899 18309 webserver.cc:460] Webserver started at http://127.0.98.3:38733/ using document root <none> and password file <none>
I20260619 12:33:56.614472 18309 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:56.614534 18309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:56.614734 18309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:33:56.616724 18309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "60bd53df06bd4ea3b1114be4c7dcab99"
format_stamp: "Formatted at 2026-06-19 12:33:56 on dist-test-slave-fg3h"
server_key: "85135b9af1ed66baebda109bf7acf818"
server_key_iv: "4e03b382e3d34258db7096ff6192259c"
server_key_version: "encryptionkey@0"
I20260619 12:33:56.617322 18309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "60bd53df06bd4ea3b1114be4c7dcab99"
format_stamp: "Formatted at 2026-06-19 12:33:56 on dist-test-slave-fg3h"
server_key: "85135b9af1ed66baebda109bf7acf818"
server_key_iv: "4e03b382e3d34258db7096ff6192259c"
server_key_version: "encryptionkey@0"
I20260619 12:33:56.621182 18309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 12:33:56.623692 18323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:56.624904 18309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:33:56.625032 18309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "60bd53df06bd4ea3b1114be4c7dcab99"
format_stamp: "Formatted at 2026-06-19 12:33:56 on dist-test-slave-fg3h"
server_key: "85135b9af1ed66baebda109bf7acf818"
server_key_iv: "4e03b382e3d34258db7096ff6192259c"
server_key_version: "encryptionkey@0"
I20260619 12:33:56.625203 18309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:56.637917 18309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:56.638574 18309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:56.638762 18309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:56.639313 18309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:33:56.640221 18309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:33:56.640291 18309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:56.640370 18309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:33:56.640420 18309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:56.649857 18309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35707
I20260619 12:33:56.649984 18436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35707 every 8 connection(s)
I20260619 12:33:56.650786 18309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:33:56.652381 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 18309
I20260619 12:33:56.652508 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:33:56.652830 392 external_mini_cluster.cc:1428] Setting key af3971b0dbc74c90c1f03ab1dd86d232
I20260619 12:33:56.660012 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:56.652629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57251 (local address 127.0.98.62:41883)
0619 12:33:56.652787 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:56.652792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:56.653967 (+ 1175us) server_negotiation.cc:407] Connection header received
0619 12:33:56.655428 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:56.655430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:56.655475 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:56.655544 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:56.656362 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.656909 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.658080 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.658337 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.658430 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:56.659509 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:56.659512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:56.659514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:33:56.659526 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:56.659545 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:33:56.659553 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:56.659625 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:56.659843 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 12:33:56.659900 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 12:33:56.660897 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:56.653068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41883 (local address 127.0.98.3:57251)
0619 12:33:56.653827 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:56.653878 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:56.655236 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:56.655671 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:56.655677 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:56.655762 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:33:56.656218 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:56.656230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.657020 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.657022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:56.657956 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:56.657965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.658468 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.658471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:56.658611 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:56.659262 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:33:56.659263 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:33:56.659303 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:33:56.659371 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:33:56.659668 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:33:56.659676 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:33:56.659681 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:33:56.659801 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 12:33:56.660015 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":137,"threads_started":1}
I20260619 12:33:56.662112 18437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41883
I20260619 12:33:56.662318 18437 heartbeater.cc:464] Registering TS with master...
I20260619 12:33:56.662774 18437 heartbeater.cc:511] Master 127.0.98.62:41883 requested a full tablet report, sending...
I20260619 12:33:56.663796 17987 ts_manager.cc:194] Registered new tserver with Master: 60bd53df06bd4ea3b1114be4c7dcab99 (127.0.98.3:35707)
I20260619 12:33:56.664425 17987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57251
I20260619 12:33:56.666673 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:33:56.672014 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:56.668579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.0.98.62:41883)
0619 12:33:56.668726 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:56.668732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:56.668867 (+ 135us) server_negotiation.cc:407] Connection header received
0619 12:33:56.668964 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:56.668966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:56.669002 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:56.669063 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:56.669347 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.669836 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.670622 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.670827 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.670909 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:56.671635 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:56.671638 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:56.671640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:33:56.671650 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:56.671666 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:33:56.671673 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:56.671731 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:56.671801 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 12:33:56.671847 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 12:33:56.674688 17987 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:33:56.677749 17987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:33:56.691850 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:56.686303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46688 (local address 127.0.98.1:40183)
0619 12:33:56.687242 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:56.687254 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:56.687279 (+ 25us) server_negotiation.cc:407] Connection header received
0619 12:33:56.687356 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:56.687362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:56.687561 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:56.687956 (+ 395us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:33:56.688099 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.688940 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.690605 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.691221 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.691357 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:56.691554 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:56.691606 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 12:33:56.691692 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":122,"threads_started":1}
I20260619 12:33:56.691859 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:56.686122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40183 (local address 127.0.0.1:46688)
0619 12:33:56.686779 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:56.686829 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:56.686953 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:56.687736 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:56.687742 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:56.687763 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:33:56.687982 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:56.687993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.689064 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.689067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:56.690496 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:56.690506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.691348 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.691351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:56.691453 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:56.691486 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 12:33:56.691582 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 12:33:56.691691 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":95,"thread_start_us":74,"threads_started":1}
I20260619 12:33:56.692307 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:56.686563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35707 (local address 127.0.0.1:60876)
0619 12:33:56.687435 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:56.687450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:56.687532 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:56.688153 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:56.688156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:56.688171 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:33:56.688258 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:56.688264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.689596 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.689602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:56.690667 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:56.690676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.691826 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.691830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:56.692009 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:56.692031 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:33:56.692095 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 12:33:56.692157 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":64,"thread_start_us":85,"threads_started":1}
I20260619 12:33:56.692596 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:56.686872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48224 (local address 127.0.98.2:45001)
0619 12:33:56.687338 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:56.687348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:56.687379 (+ 31us) server_negotiation.cc:407] Connection header received
0619 12:33:56.687441 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:56.687445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:56.687583 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:56.687720 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:33:56.688122 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.689336 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.690711 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.691497 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.691601 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:56.691762 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:56.692351 (+ 589us) server_negotiation.cc:299] Negotiation successful
0619 12:33:56.692445 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":105,"threads_started":1}
I20260619 12:33:56.692839 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:56.686936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60876 (local address 127.0.98.3:35707)
0619 12:33:56.687235 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:56.687248 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:56.687619 (+ 371us) server_negotiation.cc:407] Connection header received
0619 12:33:56.687698 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:56.687703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:56.687895 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:56.688044 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:33:56.688381 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.689484 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.690765 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.691497 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.691601 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:56.691762 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:56.692560 (+ 798us) server_negotiation.cc:299] Negotiation successful
0619 12:33:56.692672 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":85,"threads_started":1}
I20260619 12:33:56.692504 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:56.686299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45001 (local address 127.0.0.1:48224)
0619 12:33:56.686778 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:56.686830 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:56.686953 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:56.687862 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:56.687865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:56.687878 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:33:56.688019 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:56.688027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.689448 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.689451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:56.690591 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:56.690599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.691826 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.691830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:56.692205 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:56.692225 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 12:33:56.692294 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 12:33:56.692377 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":254,"spinlock_wait_cycles":3968,"thread_start_us":122,"threads_started":1}
I20260619 12:33:56.694806 18109 tablet_service.cc:1467] Processing CreateTablet for tablet 04cf4ee3b0a748ecade818679f7b81fd (DEFAULT_TABLE table=test-table [id=5eba947f799b4ff9894e54bc8579c818]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:33:56.695122 18371 tablet_service.cc:1467] Processing CreateTablet for tablet 04cf4ee3b0a748ecade818679f7b81fd (DEFAULT_TABLE table=test-table [id=5eba947f799b4ff9894e54bc8579c818]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:33:56.695757 18109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04cf4ee3b0a748ecade818679f7b81fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:33:56.695873 18371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04cf4ee3b0a748ecade818679f7b81fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:33:56.700150 18240 tablet_service.cc:1467] Processing CreateTablet for tablet 04cf4ee3b0a748ecade818679f7b81fd (DEFAULT_TABLE table=test-table [id=5eba947f799b4ff9894e54bc8579c818]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:33:56.701222 18240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04cf4ee3b0a748ecade818679f7b81fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:33:56.702590 18455 tablet_bootstrap.cc:492] T 04cf4ee3b0a748ecade818679f7b81fd P 8a4064b1f2b44501b0f450fae169be0f: Bootstrap starting.
I20260619 12:33:56.703377 18454 tablet_bootstrap.cc:492] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99: Bootstrap starting.
I20260619 12:33:56.704954 18455 tablet_bootstrap.cc:654] T 04cf4ee3b0a748ecade818679f7b81fd P 8a4064b1f2b44501b0f450fae169be0f: Neither blocks nor log segments found. Creating new log.
I20260619 12:33:56.705426 18454 tablet_bootstrap.cc:654] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99: Neither blocks nor log segments found. Creating new log.
I20260619 12:33:56.705861 18455 log.cc:824] T 04cf4ee3b0a748ecade818679f7b81fd P 8a4064b1f2b44501b0f450fae169be0f: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:56.706187 18454 log.cc:824] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:56.707948 18455 tablet_bootstrap.cc:492] T 04cf4ee3b0a748ecade818679f7b81fd P 8a4064b1f2b44501b0f450fae169be0f: No bootstrap required, opened a new log
I20260619 12:33:56.708088 18454 tablet_bootstrap.cc:492] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99: No bootstrap required, opened a new log
I20260619 12:33:56.708150 18455 ts_tablet_manager.cc:1397] T 04cf4ee3b0a748ecade818679f7b81fd P 8a4064b1f2b44501b0f450fae169be0f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:33:56.708266 18454 ts_tablet_manager.cc:1397] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 12:33:56.709432 18458 tablet_bootstrap.cc:492] T 04cf4ee3b0a748ecade818679f7b81fd P 72f07b99c28b48aa849db764f420db7e: Bootstrap starting.
I20260619 12:33:56.711419 18455 raft_consensus.cc:348] T 04cf4ee3b0a748ecade818679f7b81fd P 8a4064b1f2b44501b0f450fae169be0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4064b1f2b44501b0f450fae169be0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40183 } } peers { permanent_uuid: "72f07b99c28b48aa849db764f420db7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "60bd53df06bd4ea3b1114be4c7dcab99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35707 } }
I20260619 12:33:56.711607 18458 tablet_bootstrap.cc:654] T 04cf4ee3b0a748ecade818679f7b81fd P 72f07b99c28b48aa849db764f420db7e: Neither blocks nor log segments found. Creating new log.
I20260619 12:33:56.711652 18455 raft_consensus.cc:374] T 04cf4ee3b0a748ecade818679f7b81fd P 8a4064b1f2b44501b0f450fae169be0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:33:56.711724 18455 raft_consensus.cc:729] T 04cf4ee3b0a748ecade818679f7b81fd P 8a4064b1f2b44501b0f450fae169be0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4064b1f2b44501b0f450fae169be0f, State: Initialized, Role: FOLLOWER
I20260619 12:33:56.712172 18455 consensus_queue.cc:260] T 04cf4ee3b0a748ecade818679f7b81fd P 8a4064b1f2b44501b0f450fae169be0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4064b1f2b44501b0f450fae169be0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40183 } } peers { permanent_uuid: "72f07b99c28b48aa849db764f420db7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "60bd53df06bd4ea3b1114be4c7dcab99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35707 } }
I20260619 12:33:56.712430 18458 log.cc:824] T 04cf4ee3b0a748ecade818679f7b81fd P 72f07b99c28b48aa849db764f420db7e: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:56.712400 18454 raft_consensus.cc:348] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4064b1f2b44501b0f450fae169be0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40183 } } peers { permanent_uuid: "72f07b99c28b48aa849db764f420db7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "60bd53df06bd4ea3b1114be4c7dcab99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35707 } }
I20260619 12:33:56.712690 18454 raft_consensus.cc:374] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:33:56.712802 18175 heartbeater.cc:503] Master 127.0.98.62:41883 was elected leader, sending a full tablet report...
I20260619 12:33:56.712793 18454 raft_consensus.cc:729] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60bd53df06bd4ea3b1114be4c7dcab99, State: Initialized, Role: FOLLOWER
I20260619 12:33:56.713191 18455 ts_tablet_manager.cc:1428] T 04cf4ee3b0a748ecade818679f7b81fd P 8a4064b1f2b44501b0f450fae169be0f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 12:33:56.713279 18454 consensus_queue.cc:260] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4064b1f2b44501b0f450fae169be0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40183 } } peers { permanent_uuid: "72f07b99c28b48aa849db764f420db7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "60bd53df06bd4ea3b1114be4c7dcab99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35707 } }
I20260619 12:33:56.713989 18437 heartbeater.cc:503] Master 127.0.98.62:41883 was elected leader, sending a full tablet report...
I20260619 12:33:56.714195 18454 ts_tablet_manager.cc:1428] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 12:33:56.714551 18458 tablet_bootstrap.cc:492] T 04cf4ee3b0a748ecade818679f7b81fd P 72f07b99c28b48aa849db764f420db7e: No bootstrap required, opened a new log
I20260619 12:33:56.714715 18458 ts_tablet_manager.cc:1397] T 04cf4ee3b0a748ecade818679f7b81fd P 72f07b99c28b48aa849db764f420db7e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:33:56.718693 18458 raft_consensus.cc:348] T 04cf4ee3b0a748ecade818679f7b81fd P 72f07b99c28b48aa849db764f420db7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4064b1f2b44501b0f450fae169be0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40183 } } peers { permanent_uuid: "72f07b99c28b48aa849db764f420db7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "60bd53df06bd4ea3b1114be4c7dcab99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35707 } }
I20260619 12:33:56.718883 18458 raft_consensus.cc:374] T 04cf4ee3b0a748ecade818679f7b81fd P 72f07b99c28b48aa849db764f420db7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:33:56.718914 18458 raft_consensus.cc:729] T 04cf4ee3b0a748ecade818679f7b81fd P 72f07b99c28b48aa849db764f420db7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f07b99c28b48aa849db764f420db7e, State: Initialized, Role: FOLLOWER
I20260619 12:33:56.719277 18458 consensus_queue.cc:260] T 04cf4ee3b0a748ecade818679f7b81fd P 72f07b99c28b48aa849db764f420db7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4064b1f2b44501b0f450fae169be0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40183 } } peers { permanent_uuid: "72f07b99c28b48aa849db764f420db7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "60bd53df06bd4ea3b1114be4c7dcab99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35707 } }
I20260619 12:33:56.719863 18306 heartbeater.cc:503] Master 127.0.98.62:41883 was elected leader, sending a full tablet report...
I20260619 12:33:56.720110 18458 ts_tablet_manager.cc:1428] T 04cf4ee3b0a748ecade818679f7b81fd P 72f07b99c28b48aa849db764f420db7e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260619 12:33:56.720607 18307 tablet.cc:2369] T 04cf4ee3b0a748ecade818679f7b81fd P 72f07b99c28b48aa849db764f420db7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:33:56.781145 18176 tablet.cc:2369] T 04cf4ee3b0a748ecade818679f7b81fd P 8a4064b1f2b44501b0f450fae169be0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:33:56.901798 18438 tablet.cc:2369] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:33:56.921473 18461 raft_consensus.cc:482] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:33:56.921696 18461 raft_consensus.cc:504] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4064b1f2b44501b0f450fae169be0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40183 } } peers { permanent_uuid: "72f07b99c28b48aa849db764f420db7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "60bd53df06bd4ea3b1114be4c7dcab99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35707 } }
I20260619 12:33:56.922973 18461 leader_election.cc:288] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a4064b1f2b44501b0f450fae169be0f (127.0.98.1:40183), 72f07b99c28b48aa849db764f420db7e (127.0.98.2:45001)
I20260619 12:33:56.927847 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:56.923479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40537 (local address 127.0.98.2:45001)
0619 12:33:56.923628 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:56.923634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:56.923763 (+ 129us) server_negotiation.cc:407] Connection header received
0619 12:33:56.923847 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:56.923849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:56.923890 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:56.923958 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:33:56.924224 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.924962 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.926590 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.927205 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.927323 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:56.927381 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:56.927640 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 12:33:56.927714 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 12:33:56.927848 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:56.923266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37367 (local address 127.0.98.1:40183)
0619 12:33:56.923459 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:56.923464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:56.923481 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:33:56.923539 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:56.923542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:56.923595 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:56.923692 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:33:56.924047 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.924833 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.926589 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.927205 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.927319 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:56.927381 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:56.927640 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 12:33:56.927712 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260619 12:33:56.927878 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:56.923157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40183 (local address 127.0.98.3:37367)
0619 12:33:56.923271 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:56.923288 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:56.923390 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:56.923720 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:56.923724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:56.923747 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:33:56.923892 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:56.923898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.924950 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.924953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:56.926441 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:56.926453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.927337 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.927341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:56.927523 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:56.927540 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 12:33:56.927611 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 12:33:56.927722 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":190}
I20260619 12:33:56.927886 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:56.923269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45001 (local address 127.0.98.3:40537)
0619 12:33:56.923642 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:56.923677 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:56.923763 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:56.923999 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:56.924002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:56.924023 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:33:56.924123 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:56.924127 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.925067 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.925070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:56.926441 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:56.926453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.927337 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.927341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:56.927465 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:56.927531 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 12:33:56.927611 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 12:33:56.927722 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":93,"thread_start_us":114,"threads_started":1}
I20260619 12:33:56.928625 18260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04cf4ee3b0a748ecade818679f7b81fd" candidate_uuid: "60bd53df06bd4ea3b1114be4c7dcab99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f07b99c28b48aa849db764f420db7e" is_pre_election: true
I20260619 12:33:56.928633 18129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04cf4ee3b0a748ecade818679f7b81fd" candidate_uuid: "60bd53df06bd4ea3b1114be4c7dcab99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4064b1f2b44501b0f450fae169be0f" is_pre_election: true
I20260619 12:33:56.928920 18129 raft_consensus.cc:2454] T 04cf4ee3b0a748ecade818679f7b81fd P 8a4064b1f2b44501b0f450fae169be0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60bd53df06bd4ea3b1114be4c7dcab99 in term 0.
I20260619 12:33:56.928941 18260 raft_consensus.cc:2454] T 04cf4ee3b0a748ecade818679f7b81fd P 72f07b99c28b48aa849db764f420db7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60bd53df06bd4ea3b1114be4c7dcab99 in term 0.
I20260619 12:33:56.929461 18324 leader_election.cc:302] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60bd53df06bd4ea3b1114be4c7dcab99, 8a4064b1f2b44501b0f450fae169be0f; no voters:
I20260619 12:33:56.929745 18461 raft_consensus.cc:2781] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:33:56.929840 18461 raft_consensus.cc:482] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:33:56.929885 18461 raft_consensus.cc:3037] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:33:56.930874 18461 raft_consensus.cc:504] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4064b1f2b44501b0f450fae169be0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40183 } } peers { permanent_uuid: "72f07b99c28b48aa849db764f420db7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "60bd53df06bd4ea3b1114be4c7dcab99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35707 } }
I20260619 12:33:56.931200 18461 leader_election.cc:288] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [CANDIDATE]: Term 1 election: Requested vote from peers 8a4064b1f2b44501b0f450fae169be0f (127.0.98.1:40183), 72f07b99c28b48aa849db764f420db7e (127.0.98.2:45001)
I20260619 12:33:56.931540 18129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04cf4ee3b0a748ecade818679f7b81fd" candidate_uuid: "60bd53df06bd4ea3b1114be4c7dcab99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4064b1f2b44501b0f450fae169be0f"
I20260619 12:33:56.931535 18260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04cf4ee3b0a748ecade818679f7b81fd" candidate_uuid: "60bd53df06bd4ea3b1114be4c7dcab99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f07b99c28b48aa849db764f420db7e"
I20260619 12:33:56.931674 18129 raft_consensus.cc:3037] T 04cf4ee3b0a748ecade818679f7b81fd P 8a4064b1f2b44501b0f450fae169be0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:33:56.931686 18260 raft_consensus.cc:3037] T 04cf4ee3b0a748ecade818679f7b81fd P 72f07b99c28b48aa849db764f420db7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:33:56.932683 18260 raft_consensus.cc:2454] T 04cf4ee3b0a748ecade818679f7b81fd P 72f07b99c28b48aa849db764f420db7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60bd53df06bd4ea3b1114be4c7dcab99 in term 1.
I20260619 12:33:56.932684 18129 raft_consensus.cc:2454] T 04cf4ee3b0a748ecade818679f7b81fd P 8a4064b1f2b44501b0f450fae169be0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60bd53df06bd4ea3b1114be4c7dcab99 in term 1.
I20260619 12:33:56.932977 18324 leader_election.cc:302] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60bd53df06bd4ea3b1114be4c7dcab99, 8a4064b1f2b44501b0f450fae169be0f; no voters:
I20260619 12:33:56.933221 18461 raft_consensus.cc:2781] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:33:56.933528 18461 raft_consensus.cc:686] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [term 1 LEADER]: Becoming Leader. State: Replica: 60bd53df06bd4ea3b1114be4c7dcab99, State: Running, Role: LEADER
I20260619 12:33:56.933892 18461 consensus_queue.cc:237] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4064b1f2b44501b0f450fae169be0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40183 } } peers { permanent_uuid: "72f07b99c28b48aa849db764f420db7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "60bd53df06bd4ea3b1114be4c7dcab99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35707 } }
I20260619 12:33:56.936911 17986 catalog_manager.cc:5495] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60bd53df06bd4ea3b1114be4c7dcab99 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "60bd53df06bd4ea3b1114be4c7dcab99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4064b1f2b44501b0f450fae169be0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72f07b99c28b48aa849db764f420db7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60bd53df06bd4ea3b1114be4c7dcab99" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35707 } health_report { overall_health: HEALTHY } } }
I20260619 12:33:56.984184 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:56.980716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60878 (local address 127.0.98.3:35707)
0619 12:33:56.980853 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:56.980858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:56.980872 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:33:56.981012 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:56.981015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:56.981068 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:56.981238 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:33:56.981496 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.981997 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.982826 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:56.983041 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:56.983120 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:56.983382 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:33:56.983788 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:33:56.983890 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:56.983946 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 12:33:56.983989 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 12:33:56.990357 18260 raft_consensus.cc:1261] T 04cf4ee3b0a748ecade818679f7b81fd P 72f07b99c28b48aa849db764f420db7e [term 1 FOLLOWER]: Refusing update from remote peer 60bd53df06bd4ea3b1114be4c7dcab99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:33:56.990357 18129 raft_consensus.cc:1261] T 04cf4ee3b0a748ecade818679f7b81fd P 8a4064b1f2b44501b0f450fae169be0f [term 1 FOLLOWER]: Refusing update from remote peer 60bd53df06bd4ea3b1114be4c7dcab99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:33:56.991081 18461 consensus_queue.cc:1035] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72f07b99c28b48aa849db764f420db7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:33:56.991256 18464 consensus_queue.cc:1035] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4064b1f2b44501b0f450fae169be0f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:33:56.998718 18467 mvcc.cc:205] Tried to move back new op lower bound from 7298549501904678912 to 7298549501683642368. Current Snapshot: MvccSnapshot[applied={T|T < 7298549501904678912}]
I20260619 12:33:57.007344 17986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:33:57.007495 17986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:33:57.010037 17986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a226584bc97541979f69aa8dde880822: Sending DeleteTablet for 3 replicas of tablet 04cf4ee3b0a748ecade818679f7b81fd
I20260619 12:33:57.010838 18109 tablet_service.cc:1514] Processing DeleteTablet for tablet 04cf4ee3b0a748ecade818679f7b81fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:33:57 UTC) from {username='slave'} at 127.0.0.1:46688
I20260619 12:33:57.011142 18371 tablet_service.cc:1514] Processing DeleteTablet for tablet 04cf4ee3b0a748ecade818679f7b81fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:33:57 UTC) from {username='slave'} at 127.0.0.1:60876
I20260619 12:33:57.011370 18240 tablet_service.cc:1514] Processing DeleteTablet for tablet 04cf4ee3b0a748ecade818679f7b81fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:33:57 UTC) from {username='slave'} at 127.0.0.1:48224
I20260619 12:33:57.011627 18477 tablet_replica.cc:330] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99: stopping tablet replica
I20260619 12:33:57.011942 18477 raft_consensus.cc:2229] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:33:57.011987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 18047
I20260619 12:33:57.012002 18476 tablet_replica.cc:330] T 04cf4ee3b0a748ecade818679f7b81fd P 8a4064b1f2b44501b0f450fae169be0f: stopping tablet replica
I20260619 12:33:57.012353 18477 raft_consensus.cc:2258] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:33:57.013854 18477 ts_tablet_manager.cc:1905] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:33:57.015471 18478 tablet_replica.cc:330] T 04cf4ee3b0a748ecade818679f7b81fd P 72f07b99c28b48aa849db764f420db7e: stopping tablet replica
I20260619 12:33:57.015821 18478 raft_consensus.cc:2229] T 04cf4ee3b0a748ecade818679f7b81fd P 72f07b99c28b48aa849db764f420db7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:33:57.016167 18478 raft_consensus.cc:2258] T 04cf4ee3b0a748ecade818679f7b81fd P 72f07b99c28b48aa849db764f420db7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:33:57.016984 18477 ts_tablet_manager.cc:1918] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:33:57.017161 18477 log.cc:1196] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/04cf4ee3b0a748ecade818679f7b81fd
I20260619 12:33:57.017561 18477 ts_tablet_manager.cc:1939] T 04cf4ee3b0a748ecade818679f7b81fd P 60bd53df06bd4ea3b1114be4c7dcab99: Deleting consensus metadata
I20260619 12:33:57.017846 18478 ts_tablet_manager.cc:1905] T 04cf4ee3b0a748ecade818679f7b81fd P 72f07b99c28b48aa849db764f420db7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:33:57.018944 17973 catalog_manager.cc:4841] TS 60bd53df06bd4ea3b1114be4c7dcab99 (127.0.98.3:35707): tablet 04cf4ee3b0a748ecade818679f7b81fd (table test-table [id=5eba947f799b4ff9894e54bc8579c818]) successfully deleted
W20260619 12:33:57.020437 17971 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40183 (error 108)
I20260619 12:33:57.021503 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:57.021194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40183 (local address 127.0.0.1:46704)
0619 12:33:57.021329 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:57.021410 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 12:33:57.021569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 18178
W20260619 12:33:57.021799 17971 catalog_manager.cc:4584] TS 8a4064b1f2b44501b0f450fae169be0f (127.0.98.1:40183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 04cf4ee3b0a748ecade818679f7b81fd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40183: connect: Connection refused (error 111)
I20260619 12:33:57.034409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 18309
I20260619 12:33:57.035167 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:57.034838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45001 (local address 127.0.0.1:48236)
0619 12:33:57.035030 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:57.035081 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 12:33:57.040964 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 17956
2026-06-19T12:33:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1220 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:33:57 dist-test-slave-fg3h krb5kdc[18482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:33:57 dist-test-slave-fg3h krb5kdc[18482](info): set up 2 sockets
Jun 19 12:33:57 dist-test-slave-fg3h krb5kdc[18482](info): commencing operation
krb5kdc: starting...
W20260619 12:33:58.088989 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:33:58 dist-test-slave-fg3h krb5kdc[18482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:33:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:33:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:33:58.217631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36161
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:33:58.330053 18495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:58.330368 18495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:58.330452 18495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:58.334568 18495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:33:58.334651 18495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:58.334686 18495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:58.334728 18495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:33:58.334764 18495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:33:58.340163 18495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:58.341259 18495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:58.342312 18495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:58.351083 18501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:58.351101 18500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:58.351094 18503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:58.351346 18495 server_base.cc:1034] running on GCE node
I20260619 12:33:58.352010 18495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:58.353159 18495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:58.354363 18495 hybrid_clock.cc:648] HybridClock initialized: now 1781872438354328 us; error 47 us; skew 500 ppm
Jun 19 12:33:58 dist-test-slave-fg3h krb5kdc[18482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:33:58.361773 18495 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:33:58.362869 18495 webserver.cc:460] Webserver started at http://127.0.98.62:42743/ using document root <none> and password file <none>
I20260619 12:33:58.363535 18495 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:58.363602 18495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:58.363832 18495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:33:58.366050 18495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "c1422cc710e6491ebcb9fc6923a042c5"
format_stamp: "Formatted at 2026-06-19 12:33:58 on dist-test-slave-fg3h"
server_key: "9c64a731d144d438356a49eeb5cb2bdc"
server_key_iv: "b8d6233b7c54163c877eab4d25ac799d"
server_key_version: "encryptionkey@0"
I20260619 12:33:58.366631 18495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "c1422cc710e6491ebcb9fc6923a042c5"
format_stamp: "Formatted at 2026-06-19 12:33:58 on dist-test-slave-fg3h"
server_key: "9c64a731d144d438356a49eeb5cb2bdc"
server_key_iv: "b8d6233b7c54163c877eab4d25ac799d"
server_key_version: "encryptionkey@0"
I20260619 12:33:58.370921 18495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 12:33:58.373505 18510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:58.374778 18495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:33:58.374924 18495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "c1422cc710e6491ebcb9fc6923a042c5"
format_stamp: "Formatted at 2026-06-19 12:33:58 on dist-test-slave-fg3h"
server_key: "9c64a731d144d438356a49eeb5cb2bdc"
server_key_iv: "b8d6233b7c54163c877eab4d25ac799d"
server_key_version: "encryptionkey@0"
I20260619 12:33:58.375039 18495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:58.392378 18495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:58.395439 18495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:58.395633 18495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:58.402899 18495 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36161
I20260619 12:33:58.402923 18562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36161 every 8 connection(s)
I20260619 12:33:58.404017 18495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:33:58.406971 18563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:33:58.413018 18563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5: Bootstrap starting.
I20260619 12:33:58.414275 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 18495
I20260619 12:33:58.414434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:33:58.414819 392 external_mini_cluster.cc:1428] Setting key b64e8d1bfb6efe121f4063c49fe101f6
I20260619 12:33:58.415948 18563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5: Neither blocks nor log segments found. Creating new log.
I20260619 12:33:58.416857 18563 log.cc:824] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:58.418926 18563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5: No bootstrap required, opened a new log
I20260619 12:33:58.422850 18563 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1422cc710e6491ebcb9fc6923a042c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36161 } }
I20260619 12:33:58.423132 18563 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:33:58.423242 18563 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1422cc710e6491ebcb9fc6923a042c5, State: Initialized, Role: FOLLOWER
I20260619 12:33:58.423777 18563 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1422cc710e6491ebcb9fc6923a042c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36161 } }
I20260619 12:33:58.423956 18563 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:33:58.424054 18563 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:33:58.424198 18563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 12:33:58 dist-test-slave-fg3h krb5kdc[18482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:33:58.425370 18563 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1422cc710e6491ebcb9fc6923a042c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36161 } }
I20260619 12:33:58.425819 18563 leader_election.cc:302] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1422cc710e6491ebcb9fc6923a042c5; no voters:
I20260619 12:33:58.426246 18563 leader_election.cc:288] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:33:58.426340 18568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:33:58.426625 18568 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5 [term 1 LEADER]: Becoming Leader. State: Replica: c1422cc710e6491ebcb9fc6923a042c5, State: Running, Role: LEADER
I20260619 12:33:58.427057 18568 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1422cc710e6491ebcb9fc6923a042c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36161 } }
I20260619 12:33:58.427788 18563 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:33:58.428311 18570 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1422cc710e6491ebcb9fc6923a042c5. Latest consensus state: current_term: 1 leader_uuid: "c1422cc710e6491ebcb9fc6923a042c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1422cc710e6491ebcb9fc6923a042c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36161 } } }
I20260619 12:33:58.428437 18570 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5 [sys.catalog]: This master's current role is: LEADER
I20260619 12:33:58.429168 18569 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1422cc710e6491ebcb9fc6923a042c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1422cc710e6491ebcb9fc6923a042c5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36161 } } }
I20260619 12:33:58.429275 18569 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5 [sys.catalog]: This master's current role is: LEADER
I20260619 12:33:58.429601 18572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:33:58.431850 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:58.416091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52308 (local address 127.0.98.62:36161)
0619 12:33:58.416574 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:58.416590 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:58.416627 (+ 37us) server_negotiation.cc:407] Connection header received
0619 12:33:58.417527 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:58.417548 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:58.417935 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:58.418322 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:58.419274 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:58.420370 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:58.421413 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:58.421940 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:58.422130 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:58.425114 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:58.425122 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:58.425140 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:33:58.425179 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:58.427377 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:58.427982 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:58.427989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:58.427996 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:58.428162 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:58.428565 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:58.428568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:58.428570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:58.429224 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:58.429438 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:58.430326 (+ 888us) server_negotiation.cc:299] Negotiation successful
0619 12:33:58.430755 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":150,"threads_started":1}
I20260619 12:33:58.432950 18572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:33:58.438903 18572 catalog_manager.cc:1347] Generated new cluster ID: c1ac488b5de846d0899bd831617623c4
I20260619 12:33:58.438992 18572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:33:58.455471 18572 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:33:58.456853 18572 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:33:58.465230 18572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5: Generated new TSK 0
I20260619 12:33:58.466056 18572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:33:58.496529 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36161
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:33:58.611460 18589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:58.611735 18589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:58.611778 18589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:58.615913 18589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:58.615988 18589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:58.616081 18589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:33:58.621605 18589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:58.622741 18589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:58.623804 18589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:58.633797 18594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:58.633878 18595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:58.634058 18589 server_base.cc:1034] running on GCE node
W20260619 12:33:58.634047 18597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:58.634598 18589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:58.635255 18589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:58.636443 18589 hybrid_clock.cc:648] HybridClock initialized: now 1781872438636407 us; error 49 us; skew 500 ppm
Jun 19 12:33:58 dist-test-slave-fg3h krb5kdc[18482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:33:58.639250 18589 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:33:58.640097 18589 webserver.cc:460] Webserver started at http://127.0.98.1:41417/ using document root <none> and password file <none>
I20260619 12:33:58.640623 18589 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:58.640681 18589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:58.640843 18589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:33:58.642968 18589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "d1af1e91853c4e5bb145609c77bf5cc8"
format_stamp: "Formatted at 2026-06-19 12:33:58 on dist-test-slave-fg3h"
server_key: "2fa915c148de6ab5eeafddecdfaab701"
server_key_iv: "dc7ab11c814bcc4b4a2286806b11d7d8"
server_key_version: "encryptionkey@0"
I20260619 12:33:58.643481 18589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d1af1e91853c4e5bb145609c77bf5cc8"
format_stamp: "Formatted at 2026-06-19 12:33:58 on dist-test-slave-fg3h"
server_key: "2fa915c148de6ab5eeafddecdfaab701"
server_key_iv: "dc7ab11c814bcc4b4a2286806b11d7d8"
server_key_version: "encryptionkey@0"
I20260619 12:33:58.647328 18589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:33:58.649987 18604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:58.651369 18589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:33:58.651515 18589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "d1af1e91853c4e5bb145609c77bf5cc8"
format_stamp: "Formatted at 2026-06-19 12:33:58 on dist-test-slave-fg3h"
server_key: "2fa915c148de6ab5eeafddecdfaab701"
server_key_iv: "dc7ab11c814bcc4b4a2286806b11d7d8"
server_key_version: "encryptionkey@0"
I20260619 12:33:58.651643 18589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:58.666316 18589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:58.669764 18589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:58.669978 18589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:58.670615 18589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:33:58.671471 18589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:33:58.671541 18589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:58.671613 18589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:33:58.671664 18589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:58.680898 18589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41653
I20260619 12:33:58.680925 18717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41653 every 8 connection(s)
I20260619 12:33:58.681867 18589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:33:58.683307 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 18589
I20260619 12:33:58.683451 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:33:58.683808 392 external_mini_cluster.cc:1428] Setting key 05833feb62f4409fc485f7c6f5809d2b
Jun 19 12:33:58 dist-test-slave-fg3h krb5kdc[18482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:33:58.697849 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:58.683706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37349 (local address 127.0.98.62:36161)
0619 12:33:58.683883 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:58.683888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:58.684860 (+ 972us) server_negotiation.cc:407] Connection header received
0619 12:33:58.686272 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:58.686276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:58.686326 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:58.686407 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:58.688047 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:58.688600 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:58.690545 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:58.690798 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:58.690904 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:58.693979 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:58.693982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:58.693984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:33:58.694010 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:58.695580 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:58.696157 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:58.696161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:58.696162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:58.696237 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:58.696618 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:58.696622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:58.696623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:58.696820 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:58.696928 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:58.697528 (+ 600us) server_negotiation.cc:299] Negotiation successful
0619 12:33:58.697662 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 12:33:58.698734 18720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:58.684174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36161 (local address 127.0.98.1:37349)
0619 12:33:58.684718 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:58.684765 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:58.686062 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:58.686581 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:58.686588 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:58.687128 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:33:58.687860 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:58.687883 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:58.688757 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:58.688764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:58.690403 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:58.690418 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:58.690994 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:58.691001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:58.691245 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:58.691962 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:33:58.691984 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:33:58.693781 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:33:58.695707 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:58.695714 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:58.695730 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:58.696007 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:58.696339 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:58.696343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:58.696345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:58.696514 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:58.696928 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:33:58.696936 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:33:58.697274 (+ 338us) client_negotiation.cc:769] Sending connection context
0619 12:33:58.697473 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 12:33:58.697811 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":141,"threads_started":1}
I20260619 12:33:58.700085 18718 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36161
I20260619 12:33:58.700448 18718 heartbeater.cc:464] Registering TS with master...
I20260619 12:33:58.701169 18718 heartbeater.cc:511] Master 127.0.98.62:36161 requested a full tablet report, sending...
I20260619 12:33:58.703075 18527 ts_manager.cc:194] Registered new tserver with Master: d1af1e91853c4e5bb145609c77bf5cc8 (127.0.98.1:41653)
I20260619 12:33:58.704802 18527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:33:58.716061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36161
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:33:58.830698 18723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:58.830958 18723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:58.831002 18723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:58.835224 18723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:58.835287 18723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:58.835392 18723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:33:58.840786 18723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:58.841966 18723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:58.843041 18723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:58.853433 18729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:58.853533 18723 server_base.cc:1034] running on GCE node
W20260619 12:33:58.853545 18728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:58.853448 18731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:58.854204 18723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:58.854871 18723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:58.856112 18723 hybrid_clock.cc:648] HybridClock initialized: now 1781872438856072 us; error 52 us; skew 500 ppm
Jun 19 12:33:58 dist-test-slave-fg3h krb5kdc[18482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:33:58.859174 18723 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:33:58.860093 18723 webserver.cc:460] Webserver started at http://127.0.98.2:43069/ using document root <none> and password file <none>
I20260619 12:33:58.860641 18723 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:58.860703 18723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:58.860894 18723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:33:58.863176 18723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "79252c6d2e48477aa98055187e447a7a"
format_stamp: "Formatted at 2026-06-19 12:33:58 on dist-test-slave-fg3h"
server_key: "5f5d837ca6e9f7a1cbda7a9c6b191d74"
server_key_iv: "adf5e52ef22742667ff8164ec3691a48"
server_key_version: "encryptionkey@0"
I20260619 12:33:58.863727 18723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "79252c6d2e48477aa98055187e447a7a"
format_stamp: "Formatted at 2026-06-19 12:33:58 on dist-test-slave-fg3h"
server_key: "5f5d837ca6e9f7a1cbda7a9c6b191d74"
server_key_iv: "adf5e52ef22742667ff8164ec3691a48"
server_key_version: "encryptionkey@0"
I20260619 12:33:58.867786 18723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:33:58.870375 18738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:58.871702 18723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:33:58.871802 18723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "79252c6d2e48477aa98055187e447a7a"
format_stamp: "Formatted at 2026-06-19 12:33:58 on dist-test-slave-fg3h"
server_key: "5f5d837ca6e9f7a1cbda7a9c6b191d74"
server_key_iv: "adf5e52ef22742667ff8164ec3691a48"
server_key_version: "encryptionkey@0"
I20260619 12:33:58.871917 18723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:58.908859 18723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:58.912038 18723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:58.912194 18723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:58.912768 18723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:33:58.913640 18723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:33:58.913687 18723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:58.913728 18723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:33:58.913743 18723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:58.923884 18723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39583
I20260619 12:33:58.923902 18851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39583 every 8 connection(s)
I20260619 12:33:58.924798 18723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:33:58.933256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 18723
I20260619 12:33:58.933454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:33:58.933810 392 external_mini_cluster.cc:1428] Setting key 7577a9568cc3dd8be1f050b64133375e
Jun 19 12:33:58 dist-test-slave-fg3h krb5kdc[18482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:33:58.940448 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:58.926787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36929 (local address 127.0.98.62:36161)
0619 12:33:58.926933 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:58.926936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:58.928053 (+ 1117us) server_negotiation.cc:407] Connection header received
0619 12:33:58.929260 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:58.929264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:58.929313 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:58.929420 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:58.930792 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:58.931449 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:58.932765 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:58.932983 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:58.933155 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:58.936005 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:58.936009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:58.936011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:33:58.936037 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:58.937895 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:58.938436 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:58.938442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:58.938447 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:58.938553 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:58.939005 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:58.939012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:58.939017 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:58.939304 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:58.939426 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:58.940085 (+ 659us) server_negotiation.cc:299] Negotiation successful
0619 12:33:58.940241 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 12:33:58.941421 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:58.927093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36161 (local address 127.0.98.2:36929)
0619 12:33:58.927917 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:58.927964 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:58.929033 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:58.929558 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:58.929565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:58.930064 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:33:58.930620 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:58.930636 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:58.931581 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:58.931584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:58.932559 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:58.932569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:58.933200 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:58.933204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:58.933419 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:58.934167 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:33:58.934217 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:33:58.935813 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:33:58.938032 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:58.938037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:58.938050 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:58.938326 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:58.938734 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:58.938736 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:58.938738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:58.938893 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:58.939452 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:33:58.939457 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:33:58.939742 (+ 285us) client_negotiation.cc:769] Sending connection context
0619 12:33:58.940001 (+ 259us) client_negotiation.cc:240] Negotiation successful
0619 12:33:58.940326 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":104,"threads_started":1}
I20260619 12:33:58.942801 18852 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36161
I20260619 12:33:58.943111 18852 heartbeater.cc:464] Registering TS with master...
I20260619 12:33:58.943774 18852 heartbeater.cc:511] Master 127.0.98.62:36161 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:33:58.945159 18527 ts_manager.cc:194] Registered new tserver with Master: 79252c6d2e48477aa98055187e447a7a (127.0.98.2:39583)
I20260619 12:33:58.945786 18527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:33:58.964833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36161
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:33:59.076440 18857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:59.076696 18857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:59.076740 18857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:59.080924 18857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:59.080989 18857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:59.081082 18857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:33:59.086450 18857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:59.087497 18857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:59.088534 18857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:59.098204 18865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:59.098204 18862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:59.098243 18863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:59.098590 18857 server_base.cc:1034] running on GCE node
I20260619 12:33:59.099056 18857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:59.099705 18857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:59.100912 18857 hybrid_clock.cc:648] HybridClock initialized: now 1781872439100876 us; error 49 us; skew 500 ppm
Jun 19 12:33:59 dist-test-slave-fg3h krb5kdc[18482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:33:59.103770 18857 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:33:59.104655 18857 webserver.cc:460] Webserver started at http://127.0.98.3:33977/ using document root <none> and password file <none>
I20260619 12:33:59.105330 18857 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:59.105429 18857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:59.105662 18857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:33:59.107692 18857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "42a5a4116da7479fb9609a0e0c903b5e"
format_stamp: "Formatted at 2026-06-19 12:33:59 on dist-test-slave-fg3h"
server_key: "59d700179dc8d4ce4e8e86ac1548a153"
server_key_iv: "9dc19e775ceeb0a65e4d7f9ca86e9086"
server_key_version: "encryptionkey@0"
I20260619 12:33:59.108279 18857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "42a5a4116da7479fb9609a0e0c903b5e"
format_stamp: "Formatted at 2026-06-19 12:33:59 on dist-test-slave-fg3h"
server_key: "59d700179dc8d4ce4e8e86ac1548a153"
server_key_iv: "9dc19e775ceeb0a65e4d7f9ca86e9086"
server_key_version: "encryptionkey@0"
I20260619 12:33:59.111971 18857 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 12:33:59.114472 18872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:59.115696 18857 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:33:59.115834 18857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "42a5a4116da7479fb9609a0e0c903b5e"
format_stamp: "Formatted at 2026-06-19 12:33:59 on dist-test-slave-fg3h"
server_key: "59d700179dc8d4ce4e8e86ac1548a153"
server_key_iv: "9dc19e775ceeb0a65e4d7f9ca86e9086"
server_key_version: "encryptionkey@0"
I20260619 12:33:59.115968 18857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:59.130568 18857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:59.133599 18857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:59.133821 18857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:59.134431 18857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:33:59.135258 18857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:33:59.135339 18857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:59.135406 18857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:33:59.135450 18857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:59.145560 18857 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41985
I20260619 12:33:59.145583 18985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41985 every 8 connection(s)
I20260619 12:33:59.146561 18857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:33:59.151568 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 18857
I20260619 12:33:59.151718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:33:59.152019 392 external_mini_cluster.cc:1428] Setting key 73fd2a3db7e2fee464a4ac863f628b79
Jun 19 12:33:59 dist-test-slave-fg3h krb5kdc[18482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:33:59.161523 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:59.148667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49469 (local address 127.0.98.62:36161)
0619 12:33:59.148835 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:59.148841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:59.149841 (+ 1000us) server_negotiation.cc:407] Connection header received
0619 12:33:59.151108 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:59.151111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:59.151159 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:59.151241 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:59.152831 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.153537 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.154821 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.155039 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.155151 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:59.157790 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:59.157793 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:59.157795 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:33:59.157820 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:59.159362 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:59.159880 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:59.159883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:59.159884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:59.159971 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:59.160284 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:59.160286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:59.160287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:59.160505 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:59.160587 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:59.161232 (+ 645us) server_negotiation.cc:299] Negotiation successful
0619 12:33:59.161401 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 12:33:59.162575 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:59.149070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36161 (local address 127.0.98.3:49469)
0619 12:33:59.149679 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:59.149724 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:59.150893 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:59.151547 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:59.151558 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:59.152071 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:33:59.152658 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:59.152671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.153663 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.153667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:59.154666 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:59.154677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.155189 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.155193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:59.155367 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:59.156079 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:33:59.156095 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:33:59.157611 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:33:59.159492 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:59.159497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:59.159509 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:59.159766 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:59.160064 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:33:59.160066 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:33:59.160067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:33:59.160198 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:33:59.160635 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:33:59.160642 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:33:59.160911 (+ 269us) client_negotiation.cc:769] Sending connection context
0619 12:33:59.161206 (+ 295us) client_negotiation.cc:240] Negotiation successful
0619 12:33:59.161579 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":111,"threads_started":1}
I20260619 12:33:59.163863 18986 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36161
I20260619 12:33:59.164083 18986 heartbeater.cc:464] Registering TS with master...
I20260619 12:33:59.164525 18986 heartbeater.cc:511] Master 127.0.98.62:36161 requested a full tablet report, sending...
I20260619 12:33:59.165625 18527 ts_manager.cc:194] Registered new tserver with Master: 42a5a4116da7479fb9609a0e0c903b5e (127.0.98.3:41985)
I20260619 12:33:59.166141 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:33:59.166384 18527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49469
I20260619 12:33:59.176082 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:59.168333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52320 (local address 127.0.98.62:36161)
0619 12:33:59.168467 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:59.168471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:59.168630 (+ 159us) server_negotiation.cc:407] Connection header received
0619 12:33:59.168719 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:59.168721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:59.168759 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:59.168836 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:59.169565 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.170072 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.170921 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.171127 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.171211 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:59.172675 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:59.172678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:59.172680 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:33:59.172702 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:59.174225 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:59.174699 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:59.174702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:59.174703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:59.174788 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:33:59.175101 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:33:59.175104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:33:59.175105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:33:59.175410 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:59.175532 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:59.175815 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 12:33:59.175915 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 12:33:59.179106 18527 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:33:59.182216 18527 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:33:59.197218 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:59.191272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37696 (local address 127.0.98.1:41653)
0619 12:33:59.192287 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:59.192303 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:59.192482 (+ 179us) server_negotiation.cc:407] Connection header received
0619 12:33:59.192705 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:59.192709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:59.192856 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:59.192964 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:33:59.193486 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.194615 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.195937 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.196479 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.196604 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:59.196758 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:59.196923 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 12:33:59.197024 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907,"thread_start_us":98,"threads_started":1}
I20260619 12:33:59.197212 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:59.191107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41653 (local address 127.0.0.1:37696)
0619 12:33:59.192400 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:59.192413 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:59.192553 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:59.193188 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:59.193190 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:59.193206 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:33:59.193306 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:59.193310 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.194745 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.194749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:59.195821 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:59.195828 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.196604 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.196608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:59.196709 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:59.196744 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 12:33:59.196892 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 12:33:59.197029 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1233,"mutex_wait_us":538,"thread_start_us":90,"threads_started":1}
I20260619 12:33:59.197434 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:59.191476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39583 (local address 127.0.0.1:37980)
0619 12:33:59.191961 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:59.192018 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:59.192159 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:59.192957 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:59.192964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:59.192997 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:33:59.193291 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:59.193301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.194320 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.194323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:59.195559 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:59.195570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.196829 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.196833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:59.196937 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:59.196967 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 12:33:59.197031 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 12:33:59.197140 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":78,"spinlock_wait_cycles":11776,"thread_start_us":85,"threads_started":1}
I20260619 12:33:59.197554 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:59.191921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37980 (local address 127.0.98.2:39583)
0619 12:33:59.192494 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:59.192508 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:59.192528 (+ 20us) server_negotiation.cc:407] Connection header received
0619 12:33:59.192601 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:59.192609 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:59.192812 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:59.192964 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:33:59.193418 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.194178 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.195695 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.196505 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.196608 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:59.196761 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:59.197291 (+ 530us) server_negotiation.cc:299] Negotiation successful
0619 12:33:59.197394 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":93,"threads_started":1}
I20260619 12:33:59.197769 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:59.191400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39194 (local address 127.0.98.3:41985)
0619 12:33:59.192060 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:59.192075 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:59.192100 (+ 25us) server_negotiation.cc:407] Connection header received
0619 12:33:59.192361 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:59.192367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:59.192589 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:59.192760 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:33:59.193396 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.194590 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.196077 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.196864 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.196962 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:59.197119 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:59.197496 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 12:33:59.197597 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":104,"threads_started":1}
I20260619 12:33:59.198033 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:59.191109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41985 (local address 127.0.0.1:39194)
0619 12:33:59.191960 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:59.192019 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:59.192160 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:59.192957 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:59.192964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:59.192996 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:33:59.193254 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:59.193264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.194764 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.194767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:59.195942 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:59.195953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.197174 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.197177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:59.197278 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:59.197300 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 12:33:59.197367 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 12:33:59.197421 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":123,"spinlock_wait_cycles":10752,"thread_start_us":85,"threads_started":1}
I20260619 12:33:59.200793 18920 tablet_service.cc:1467] Processing CreateTablet for tablet ae32808987bd4232a219872a264ef997 (DEFAULT_TABLE table=test-table [id=97b05b0f247849ba98848a48f48486df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:33:59.200805 18652 tablet_service.cc:1467] Processing CreateTablet for tablet ae32808987bd4232a219872a264ef997 (DEFAULT_TABLE table=test-table [id=97b05b0f247849ba98848a48f48486df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:33:59.200809 18786 tablet_service.cc:1467] Processing CreateTablet for tablet ae32808987bd4232a219872a264ef997 (DEFAULT_TABLE table=test-table [id=97b05b0f247849ba98848a48f48486df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:33:59.201843 18920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae32808987bd4232a219872a264ef997. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:33:59.201862 18652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae32808987bd4232a219872a264ef997. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:33:59.201862 18786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae32808987bd4232a219872a264ef997. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:33:59.208195 19003 tablet_bootstrap.cc:492] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e: Bootstrap starting.
I20260619 12:33:59.209396 19004 tablet_bootstrap.cc:492] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8: Bootstrap starting.
I20260619 12:33:59.209734 19005 tablet_bootstrap.cc:492] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a: Bootstrap starting.
I20260619 12:33:59.210523 19003 tablet_bootstrap.cc:654] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e: Neither blocks nor log segments found. Creating new log.
I20260619 12:33:59.211656 19003 log.cc:824] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:59.212013 19005 tablet_bootstrap.cc:654] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a: Neither blocks nor log segments found. Creating new log.
I20260619 12:33:59.212792 19005 log.cc:824] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:59.213742 19003 tablet_bootstrap.cc:492] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e: No bootstrap required, opened a new log
I20260619 12:33:59.213925 19004 tablet_bootstrap.cc:654] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8: Neither blocks nor log segments found. Creating new log.
I20260619 12:33:59.213934 19003 ts_tablet_manager.cc:1397] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260619 12:33:59.214569 19004 log.cc:824] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:59.214723 19005 tablet_bootstrap.cc:492] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a: No bootstrap required, opened a new log
I20260619 12:33:59.214916 19005 ts_tablet_manager.cc:1397] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260619 12:33:59.216209 19004 tablet_bootstrap.cc:492] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8: No bootstrap required, opened a new log
I20260619 12:33:59.216434 19004 ts_tablet_manager.cc:1397] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 12:33:59.217921 19003 raft_consensus.cc:348] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1af1e91853c4e5bb145609c77bf5cc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } } peers { permanent_uuid: "42a5a4116da7479fb9609a0e0c903b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41985 } } peers { permanent_uuid: "79252c6d2e48477aa98055187e447a7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } }
I20260619 12:33:59.218223 19003 raft_consensus.cc:374] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:33:59.218325 19003 raft_consensus.cc:729] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a5a4116da7479fb9609a0e0c903b5e, State: Initialized, Role: FOLLOWER
I20260619 12:33:59.218782 19003 consensus_queue.cc:260] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1af1e91853c4e5bb145609c77bf5cc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } } peers { permanent_uuid: "42a5a4116da7479fb9609a0e0c903b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41985 } } peers { permanent_uuid: "79252c6d2e48477aa98055187e447a7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } }
I20260619 12:33:59.218672 19005 raft_consensus.cc:348] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1af1e91853c4e5bb145609c77bf5cc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } } peers { permanent_uuid: "42a5a4116da7479fb9609a0e0c903b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41985 } } peers { permanent_uuid: "79252c6d2e48477aa98055187e447a7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } }
I20260619 12:33:59.218931 19005 raft_consensus.cc:374] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:33:59.219044 19005 raft_consensus.cc:729] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79252c6d2e48477aa98055187e447a7a, State: Initialized, Role: FOLLOWER
I20260619 12:33:59.219463 18986 heartbeater.cc:503] Master 127.0.98.62:36161 was elected leader, sending a full tablet report...
I20260619 12:33:59.219897 19003 ts_tablet_manager.cc:1428] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 12:33:59.220592 19005 consensus_queue.cc:260] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1af1e91853c4e5bb145609c77bf5cc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } } peers { permanent_uuid: "42a5a4116da7479fb9609a0e0c903b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41985 } } peers { permanent_uuid: "79252c6d2e48477aa98055187e447a7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } }
I20260619 12:33:59.220593 19004 raft_consensus.cc:348] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1af1e91853c4e5bb145609c77bf5cc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } } peers { permanent_uuid: "42a5a4116da7479fb9609a0e0c903b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41985 } } peers { permanent_uuid: "79252c6d2e48477aa98055187e447a7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } }
I20260619 12:33:59.220867 19004 raft_consensus.cc:374] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:33:59.220925 19004 raft_consensus.cc:729] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1af1e91853c4e5bb145609c77bf5cc8, State: Initialized, Role: FOLLOWER
I20260619 12:33:59.221339 19004 consensus_queue.cc:260] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1af1e91853c4e5bb145609c77bf5cc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } } peers { permanent_uuid: "42a5a4116da7479fb9609a0e0c903b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41985 } } peers { permanent_uuid: "79252c6d2e48477aa98055187e447a7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } }
I20260619 12:33:59.221555 19005 ts_tablet_manager.cc:1428] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260619 12:33:59.221902 18852 heartbeater.cc:503] Master 127.0.98.62:36161 was elected leader, sending a full tablet report...
I20260619 12:33:59.222246 19004 ts_tablet_manager.cc:1428] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:33:59.222571 18718 heartbeater.cc:503] Master 127.0.98.62:36161 was elected leader, sending a full tablet report...
I20260619 12:33:59.275970 19011 raft_consensus.cc:482] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:33:59.276208 19011 raft_consensus.cc:504] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1af1e91853c4e5bb145609c77bf5cc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } } peers { permanent_uuid: "42a5a4116da7479fb9609a0e0c903b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41985 } } peers { permanent_uuid: "79252c6d2e48477aa98055187e447a7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } }
I20260619 12:33:59.277477 19011 leader_election.cc:288] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42a5a4116da7479fb9609a0e0c903b5e (127.0.98.3:41985), 79252c6d2e48477aa98055187e447a7a (127.0.98.2:39583)
I20260619 12:33:59.283104 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:59.277876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40935 (local address 127.0.98.2:39583)
0619 12:33:59.278015 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:59.278020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:59.278401 (+ 381us) server_negotiation.cc:407] Connection header received
0619 12:33:59.278589 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:59.278593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:59.278666 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:59.278753 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:33:59.279053 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.279891 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.281733 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.282361 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.282486 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:59.282552 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:59.282896 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 12:33:59.282969 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:33:59.283104 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:59.277967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42623 (local address 127.0.98.3:41985)
0619 12:33:59.278108 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:59.278112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:59.278471 (+ 359us) server_negotiation.cc:407] Connection header received
0619 12:33:59.278714 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:59.278720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:59.278776 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:59.278849 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:33:59.279184 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.279954 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.281733 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.282401 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.282497 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:59.282552 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:59.282896 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 12:33:59.282969 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 12:33:59.283190 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:59.277785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39583 (local address 127.0.98.1:40935)
0619 12:33:59.278289 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:59.278303 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:59.278465 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:59.278764 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:59.278766 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:59.278779 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:33:59.278935 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:59.278942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.280042 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.280047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:59.281582 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:59.281594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.282553 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.282557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:59.282723 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:59.282771 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 12:33:59.282860 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 12:33:59.282984 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":177,"thread_start_us":181,"threads_started":1}
I20260619 12:33:59.283190 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:59.277785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41985 (local address 127.0.98.1:42623)
0619 12:33:59.278333 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:59.278363 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:59.278594 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:59.278894 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:59.278898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:59.278911 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:33:59.279040 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:59.279048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.280054 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.280056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:59.281582 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:59.281594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.282553 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.282557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:59.282719 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:59.282771 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 12:33:59.282860 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 12:33:59.282984 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":181,"thread_start_us":117,"threads_started":1}
I20260619 12:33:59.284147 18806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae32808987bd4232a219872a264ef997" candidate_uuid: "d1af1e91853c4e5bb145609c77bf5cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79252c6d2e48477aa98055187e447a7a" is_pre_election: true
I20260619 12:33:59.284143 18940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae32808987bd4232a219872a264ef997" candidate_uuid: "d1af1e91853c4e5bb145609c77bf5cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a5a4116da7479fb9609a0e0c903b5e" is_pre_election: true
I20260619 12:33:59.284519 18940 raft_consensus.cc:2454] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1af1e91853c4e5bb145609c77bf5cc8 in term 0.
I20260619 12:33:59.284519 18806 raft_consensus.cc:2454] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1af1e91853c4e5bb145609c77bf5cc8 in term 0.
I20260619 12:33:59.285082 18608 leader_election.cc:302] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 42a5a4116da7479fb9609a0e0c903b5e, 79252c6d2e48477aa98055187e447a7a, d1af1e91853c4e5bb145609c77bf5cc8; no voters:
I20260619 12:33:59.285423 19011 raft_consensus.cc:2781] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:33:59.285509 19011 raft_consensus.cc:482] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:33:59.285554 19011 raft_consensus.cc:3037] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:33:59.286628 19011 raft_consensus.cc:504] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1af1e91853c4e5bb145609c77bf5cc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } } peers { permanent_uuid: "42a5a4116da7479fb9609a0e0c903b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41985 } } peers { permanent_uuid: "79252c6d2e48477aa98055187e447a7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } }
I20260619 12:33:59.286981 19011 leader_election.cc:288] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [CANDIDATE]: Term 1 election: Requested vote from peers 42a5a4116da7479fb9609a0e0c903b5e (127.0.98.3:41985), 79252c6d2e48477aa98055187e447a7a (127.0.98.2:39583)
I20260619 12:33:59.287364 18940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae32808987bd4232a219872a264ef997" candidate_uuid: "d1af1e91853c4e5bb145609c77bf5cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42a5a4116da7479fb9609a0e0c903b5e"
I20260619 12:33:59.287425 18806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae32808987bd4232a219872a264ef997" candidate_uuid: "d1af1e91853c4e5bb145609c77bf5cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79252c6d2e48477aa98055187e447a7a"
I20260619 12:33:59.287477 18940 raft_consensus.cc:3037] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:33:59.287611 18806 raft_consensus.cc:3037] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:33:59.288627 18940 raft_consensus.cc:2454] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1af1e91853c4e5bb145609c77bf5cc8 in term 1.
I20260619 12:33:59.288733 18806 raft_consensus.cc:2454] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1af1e91853c4e5bb145609c77bf5cc8 in term 1.
I20260619 12:33:59.289031 18607 leader_election.cc:302] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42a5a4116da7479fb9609a0e0c903b5e, d1af1e91853c4e5bb145609c77bf5cc8; no voters:
I20260619 12:33:59.289228 19011 raft_consensus.cc:2781] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:33:59.289458 19011 raft_consensus.cc:686] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [term 1 LEADER]: Becoming Leader. State: Replica: d1af1e91853c4e5bb145609c77bf5cc8, State: Running, Role: LEADER
I20260619 12:33:59.289798 19011 consensus_queue.cc:237] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1af1e91853c4e5bb145609c77bf5cc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } } peers { permanent_uuid: "42a5a4116da7479fb9609a0e0c903b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41985 } } peers { permanent_uuid: "79252c6d2e48477aa98055187e447a7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } }
I20260619 12:33:59.293190 18527 catalog_manager.cc:5495] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1af1e91853c4e5bb145609c77bf5cc8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d1af1e91853c4e5bb145609c77bf5cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1af1e91853c4e5bb145609c77bf5cc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42a5a4116da7479fb9609a0e0c903b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79252c6d2e48477aa98055187e447a7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 } health_report { overall_health: UNKNOWN } } }
I20260619 12:33:59.325139 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:59.321317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37712 (local address 127.0.98.1:41653)
0619 12:33:59.321501 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:59.321506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:59.321522 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:33:59.321611 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:59.321613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:59.321671 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:59.321773 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:33:59.322057 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.322596 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.323475 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.323687 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.323778 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:59.324114 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:33:59.324555 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:33:59.324668 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:59.324909 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 12:33:59.324956 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 12:33:59.331362 18806 raft_consensus.cc:1261] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a [term 1 FOLLOWER]: Refusing update from remote peer d1af1e91853c4e5bb145609c77bf5cc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:33:59.331377 18940 raft_consensus.cc:1261] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e [term 1 FOLLOWER]: Refusing update from remote peer d1af1e91853c4e5bb145609c77bf5cc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:33:59.332096 19011 consensus_queue.cc:1035] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42a5a4116da7479fb9609a0e0c903b5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:33:59.332340 19014 consensus_queue.cc:1035] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79252c6d2e48477aa98055187e447a7a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:33:59.340713 19018 mvcc.cc:205] Tried to move back new op lower bound from 7298549511493726208 to 7298549511334043648. Current Snapshot: MvccSnapshot[applied={T|T < 7298549511493726208}]
I20260619 12:33:59.342973 19019 mvcc.cc:205] Tried to move back new op lower bound from 7298549511493726208 to 7298549511334043648. Current Snapshot: MvccSnapshot[applied={T|T < 7298549511493726208}]
I20260619 12:33:59.349179 18527 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:33:59.349329 18527 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:33:59.351713 18527 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1422cc710e6491ebcb9fc6923a042c5: Sending DeleteTablet for 3 replicas of tablet ae32808987bd4232a219872a264ef997
I20260619 12:33:59.352445 18652 tablet_service.cc:1514] Processing DeleteTablet for tablet ae32808987bd4232a219872a264ef997 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:33:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37696
I20260619 12:33:59.353005 18920 tablet_service.cc:1514] Processing DeleteTablet for tablet ae32808987bd4232a219872a264ef997 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:33:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39194
I20260619 12:33:59.353511 18786 tablet_service.cc:1514] Processing DeleteTablet for tablet ae32808987bd4232a219872a264ef997 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:33:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37980
I20260619 12:33:59.353650 19028 tablet_replica.cc:330] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e: stopping tablet replica
I20260619 12:33:59.353940 19027 tablet_replica.cc:330] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8: stopping tablet replica
I20260619 12:33:59.354246 19027 raft_consensus.cc:2229] T ae32808987bd4232a219872a264ef997 P d1af1e91853c4e5bb145609c77bf5cc8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:33:59.354316 19028 raft_consensus.cc:2229] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:33:59.354542 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 18589
I20260619 12:33:59.356443 19028 raft_consensus.cc:2258] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:33:59.356854 19029 tablet_replica.cc:330] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a: stopping tablet replica
I20260619 12:33:59.357206 19029 raft_consensus.cc:2229] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:33:59.357491 19029 raft_consensus.cc:2258] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:33:59.358134 19028 ts_tablet_manager.cc:1905] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:33:59.358711 19029 ts_tablet_manager.cc:1905] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:33:59.360841 19029 ts_tablet_manager.cc:1918] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:33:59.360956 19029 log.cc:1196] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/ae32808987bd4232a219872a264ef997
I20260619 12:33:59.361387 19029 ts_tablet_manager.cc:1939] T ae32808987bd4232a219872a264ef997 P 79252c6d2e48477aa98055187e447a7a: Deleting consensus metadata
I20260619 12:33:59.361588 19028 ts_tablet_manager.cc:1918] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:33:59.361693 19028 log.cc:1196] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/ae32808987bd4232a219872a264ef997
I20260619 12:33:59.362053 19028 ts_tablet_manager.cc:1939] T ae32808987bd4232a219872a264ef997 P 42a5a4116da7479fb9609a0e0c903b5e: Deleting consensus metadata
W20260619 12:33:59.362288 18511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41653 (error 108)
I20260619 12:33:59.362829 18514 catalog_manager.cc:4841] TS 79252c6d2e48477aa98055187e447a7a (127.0.98.2:39583): tablet ae32808987bd4232a219872a264ef997 (table test-table [id=97b05b0f247849ba98848a48f48486df]) successfully deleted
I20260619 12:33:59.363226 18513 catalog_manager.cc:4841] TS 42a5a4116da7479fb9609a0e0c903b5e (127.0.98.3:41985): tablet ae32808987bd4232a219872a264ef997 (table test-table [id=97b05b0f247849ba98848a48f48486df]) successfully deleted
W20260619 12:33:59.363500 19000 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:33:59.363190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41653 (local address 127.0.0.1:37714)
0619 12:33:59.363346 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:59.363422 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41653: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 12:33:59.363679 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 18723
W20260619 12:33:59.363651 18511 catalog_manager.cc:4584] TS d1af1e91853c4e5bb145609c77bf5cc8 (127.0.98.1:41653): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae32808987bd4232a219872a264ef997: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41653: connect: Connection reset by peer (error 104)
I20260619 12:33:59.369524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 18857
I20260619 12:33:59.375535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 18495
2026-06-19T12:33:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2335 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T12:33:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:33:59Z Disabled control of system clock
I20260619 12:33:59.406555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46865
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 12:33:59.516124 19034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:59.516410 19034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:59.516448 19034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:59.520514 19034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:33:59.520578 19034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:59.520594 19034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:59.520613 19034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:33:59.520628 19034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:33:59.525696 19034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46865
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:59.526826 19034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:59.528280 19034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:33:59.537282 19034 server_base.cc:1034] running on GCE node
W20260619 12:33:59.537308 19040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:59.537320 19042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:59.537290 19039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:59.538199 19034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:59.539361 19034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:59.540571 19034 hybrid_clock.cc:648] HybridClock initialized: now 1781872439540554 us; error 31 us; skew 500 ppm
I20260619 12:33:59.542569 19034 webserver.cc:460] Webserver started at http://127.0.98.62:36341/ using document root <none> and password file <none>
I20260619 12:33:59.543161 19034 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:59.543253 19034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:59.543466 19034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:33:59.545589 19034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "f7f4530d0be347a6a0361edece5cf383"
format_stamp: "Formatted at 2026-06-19 12:33:59 on dist-test-slave-fg3h"
server_key: "8184b9a9160534dbc38e018543fe6198"
server_key_iv: "a7b30b5f4ba2fef74af4b15c3e56e30b"
server_key_version: "encryptionkey@0"
I20260619 12:33:59.546142 19034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "f7f4530d0be347a6a0361edece5cf383"
format_stamp: "Formatted at 2026-06-19 12:33:59 on dist-test-slave-fg3h"
server_key: "8184b9a9160534dbc38e018543fe6198"
server_key_iv: "a7b30b5f4ba2fef74af4b15c3e56e30b"
server_key_version: "encryptionkey@0"
I20260619 12:33:59.549942 19034 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 12:33:59.552326 19048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:59.553627 19034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 12:33:59.553782 19034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "f7f4530d0be347a6a0361edece5cf383"
format_stamp: "Formatted at 2026-06-19 12:33:59 on dist-test-slave-fg3h"
server_key: "8184b9a9160534dbc38e018543fe6198"
server_key_iv: "a7b30b5f4ba2fef74af4b15c3e56e30b"
server_key_version: "encryptionkey@0"
I20260619 12:33:59.553897 19034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:59.574227 19034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:59.574985 19034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:59.575204 19034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:59.582680 19034 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46865
I20260619 12:33:59.582710 19100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46865 every 8 connection(s)
I20260619 12:33:59.583630 19034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:33:59.586813 19101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:33:59.592496 19101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383: Bootstrap starting.
I20260619 12:33:59.592813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 19034
I20260619 12:33:59.592952 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:33:59.593277 392 external_mini_cluster.cc:1428] Setting key abae93833c2f1ef1e9a42baf69d44bb2
I20260619 12:33:59.595242 19101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383: Neither blocks nor log segments found. Creating new log.
I20260619 12:33:59.595991 19101 log.cc:824] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383: Log is configured to *not* fsync() on all Append() calls
I20260619 12:33:59.598114 19101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383: No bootstrap required, opened a new log
I20260619 12:33:59.602423 19101 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f4530d0be347a6a0361edece5cf383" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46865 } }
I20260619 12:33:59.602643 19101 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:33:59.602702 19101 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f4530d0be347a6a0361edece5cf383, State: Initialized, Role: FOLLOWER
I20260619 12:33:59.603042 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:59.594688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52696 (local address 127.0.98.62:46865)
0619 12:33:59.595341 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:59.595361 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:59.595405 (+ 44us) server_negotiation.cc:407] Connection header received
0619 12:33:59.596074 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:59.596101 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:59.596682 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:59.597021 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:59.597496 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.598550 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.599732 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.600220 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.600343 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:59.601221 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:59.601227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:59.601238 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:33:59.601614 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:59.601666 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:33:59.601684 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:59.601760 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:59.601921 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 12:33:59.602207 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":143,"threads_started":1}
I20260619 12:33:59.603183 19101 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f4530d0be347a6a0361edece5cf383" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46865 } }
I20260619 12:33:59.603361 19101 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:33:59.603498 19101 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:33:59.603602 19101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:33:59.604485 19101 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f4530d0be347a6a0361edece5cf383" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46865 } }
I20260619 12:33:59.604826 19101 leader_election.cc:302] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7f4530d0be347a6a0361edece5cf383; no voters:
I20260619 12:33:59.605182 19101 leader_election.cc:288] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:33:59.605250 19106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:33:59.605465 19106 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383 [term 1 LEADER]: Becoming Leader. State: Replica: f7f4530d0be347a6a0361edece5cf383, State: Running, Role: LEADER
I20260619 12:33:59.605762 19106 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f4530d0be347a6a0361edece5cf383" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46865 } }
I20260619 12:33:59.606458 19101 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:33:59.607414 19107 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7f4530d0be347a6a0361edece5cf383. Latest consensus state: current_term: 1 leader_uuid: "f7f4530d0be347a6a0361edece5cf383" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f4530d0be347a6a0361edece5cf383" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46865 } } }
I20260619 12:33:59.607543 19107 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383 [sys.catalog]: This master's current role is: LEADER
I20260619 12:33:59.607895 19108 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7f4530d0be347a6a0361edece5cf383" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f4530d0be347a6a0361edece5cf383" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46865 } } }
I20260619 12:33:59.608038 19108 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383 [sys.catalog]: This master's current role is: LEADER
I20260619 12:33:59.608691 19115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:33:59.611400 19115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:33:59.616950 19115 catalog_manager.cc:1347] Generated new cluster ID: 0720cc09391d48ffbd19272d04096426
I20260619 12:33:59.617026 19115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:33:59.642503 19115 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:33:59.643828 19115 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:33:59.651662 19115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383: Generated new TSK 0
I20260619 12:33:59.652470 19115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:33:59.663937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46865
--builtin_ntp_servers=127.0.98.20:45227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 12:33:59.775753 19125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:59.776016 19125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:59.776055 19125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:59.780005 19125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:59.780058 19125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:59.780112 19125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:33:59.784720 19125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:59.785727 19125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:59.787015 19125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:59.796444 19130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:59.796438 19133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:59.796437 19131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:59.796660 19125 server_base.cc:1034] running on GCE node
I20260619 12:33:59.797175 19125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:59.797830 19125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:59.799021 19125 hybrid_clock.cc:648] HybridClock initialized: now 1781872439799001 us; error 41 us; skew 500 ppm
I20260619 12:33:59.800920 19125 webserver.cc:460] Webserver started at http://127.0.98.1:45571/ using document root <none> and password file <none>
I20260619 12:33:59.801559 19125 fs_manager.cc:359] Metadata directory not provided
I20260619 12:33:59.801652 19125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:33:59.801885 19125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:33:59.803908 19125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "a303b01a765e485c85eeb1d1659241a7"
format_stamp: "Formatted at 2026-06-19 12:33:59 on dist-test-slave-fg3h"
server_key: "4b5dcd5694a60af46fcaa7448e23af9b"
server_key_iv: "baf5c1372f66220988f07eced4e32ca2"
server_key_version: "encryptionkey@0"
I20260619 12:33:59.804466 19125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a303b01a765e485c85eeb1d1659241a7"
format_stamp: "Formatted at 2026-06-19 12:33:59 on dist-test-slave-fg3h"
server_key: "4b5dcd5694a60af46fcaa7448e23af9b"
server_key_iv: "baf5c1372f66220988f07eced4e32ca2"
server_key_version: "encryptionkey@0"
I20260619 12:33:59.808406 19125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:33:59.810869 19139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:59.812075 19125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 12:33:59.812225 19125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "a303b01a765e485c85eeb1d1659241a7"
format_stamp: "Formatted at 2026-06-19 12:33:59 on dist-test-slave-fg3h"
server_key: "4b5dcd5694a60af46fcaa7448e23af9b"
server_key_iv: "baf5c1372f66220988f07eced4e32ca2"
server_key_version: "encryptionkey@0"
I20260619 12:33:59.812340 19125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:33:59.837611 19125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:33:59.838346 19125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:33:59.838567 19125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:33:59.839171 19125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:33:59.840147 19125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:33:59.840224 19125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:59.840297 19125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:33:59.840343 19125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:33:59.849499 19125 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36751
I20260619 12:33:59.849519 19252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36751 every 8 connection(s)
I20260619 12:33:59.850622 19125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:33:59.860395 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:59.852673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46219 (local address 127.0.98.62:46865)
0619 12:33:59.852851 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 12:33:59.852854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:33:59.853999 (+ 1145us) server_negotiation.cc:407] Connection header received
0619 12:33:59.855455 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:33:59.855457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:33:59.855505 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:33:59.855582 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:33:59.856529 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.857185 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.858473 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.858693 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.858796 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:59.859895 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:33:59.859898 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:33:59.859900 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:33:59.859913 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:33:59.859931 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:33:59.859938 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:33:59.860004 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 12:33:59.860208 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 12:33:59.860261 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 12:33:59.860541 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 19125
I20260619 12:33:59.860671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:33:59.860966 392 external_mini_cluster.cc:1428] Setting key 6177e77cbe8c20de45e08d6ea40985b1
I20260619 12:33:59.861630 19255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:33:59.853074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46865 (local address 127.0.98.1:46219)
0619 12:33:59.853837 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0619 12:33:59.853880 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:33:59.855231 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:33:59.855747 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:33:59.855754 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:33:59.855842 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:33:59.856367 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:59.856379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.857315 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.857318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:59.858345 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:33:59.858355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:33:59.858847 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:33:59.858850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:33:59.859002 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:33:59.859595 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:33:59.859597 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:33:59.859639 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:33:59.859728 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:33:59.860042 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:33:59.860052 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:33:59.860056 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 12:33:59.860257 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 12:33:59.860568 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":130,"threads_started":1}
I20260619 12:33:59.863000 19253 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46865
I20260619 12:33:59.863297 19253 heartbeater.cc:464] Registering TS with master...
I20260619 12:33:59.863312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46865
--builtin_ntp_servers=127.0.98.20:45227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 12:33:59.863840 19253 heartbeater.cc:511] Master 127.0.98.62:46865 requested a full tablet report, sending...
I20260619 12:33:59.865624 19061 ts_manager.cc:194] Registered new tserver with Master: a303b01a765e485c85eeb1d1659241a7 (127.0.98.1:36751)
I20260619 12:33:59.866842 19061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46219
W20260619 12:33:59.974349 19256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:33:59.974615 19256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:33:59.974654 19256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:33:59.978734 19256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:33:59.978792 19256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:33:59.978883 19256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:33:59.983685 19256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:33:59.984793 19256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:33:59.986217 19256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:33:59.995441 19261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:59.995469 19262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:33:59.995441 19264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:33:59.995523 19256 server_base.cc:1034] running on GCE node
I20260619 12:33:59.996155 19256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:33:59.996809 19256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:33:59.997992 19256 hybrid_clock.cc:648] HybridClock initialized: now 1781872439997941 us; error 63 us; skew 500 ppm
I20260619 12:33:59.999782 19256 webserver.cc:460] Webserver started at http://127.0.98.2:40341/ using document root <none> and password file <none>
I20260619 12:34:00.000525 19256 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:00.000643 19256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:00.000900 19256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:00.003115 19256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "ac3e8f2973fd423ba787b5220d4c6382"
format_stamp: "Formatted at 2026-06-19 12:33:59 on dist-test-slave-fg3h"
server_key: "8e23db9d16ff0e8c52c203fab8eff774"
server_key_iv: "540e05020a48aa49bde0cced443381c3"
server_key_version: "encryptionkey@0"
I20260619 12:34:00.003712 19256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ac3e8f2973fd423ba787b5220d4c6382"
format_stamp: "Formatted at 2026-06-19 12:33:59 on dist-test-slave-fg3h"
server_key: "8e23db9d16ff0e8c52c203fab8eff774"
server_key_iv: "540e05020a48aa49bde0cced443381c3"
server_key_version: "encryptionkey@0"
I20260619 12:34:00.007903 19256 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:34:00.010676 19270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:00.012037 19256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 12:34:00.012200 19256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "ac3e8f2973fd423ba787b5220d4c6382"
format_stamp: "Formatted at 2026-06-19 12:33:59 on dist-test-slave-fg3h"
server_key: "8e23db9d16ff0e8c52c203fab8eff774"
server_key_iv: "540e05020a48aa49bde0cced443381c3"
server_key_version: "encryptionkey@0"
I20260619 12:34:00.012385 19256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:00.022955 19256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:00.023641 19256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:00.023847 19256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:00.024457 19256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:34:00.025615 19256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:34:00.025687 19256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:00.025763 19256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:34:00.025811 19256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:00.035594 19256 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41231
I20260619 12:34:00.035615 19383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41231 every 8 connection(s)
I20260619 12:34:00.036613 19256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:34:00.039501 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 19256
I20260619 12:34:00.039608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:34:00.039990 392 external_mini_cluster.cc:1428] Setting key a409f1b73cd524a678e829d092c5dd5e
I20260619 12:34:00.042712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46865
--builtin_ntp_servers=127.0.98.20:45227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 12:34:00.048657 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:00.038622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34897 (local address 127.0.98.62:46865)
0619 12:34:00.038794 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:00.038798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:00.040469 (+ 1671us) server_negotiation.cc:407] Connection header received
0619 12:34:00.042326 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:00.042329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:00.042372 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:00.042452 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:00.043739 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.044531 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.046358 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.046578 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.046679 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:00.048139 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:00.048142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:00.048144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:00.048169 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:00.048188 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:00.048197 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:00.048265 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:00.048476 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 12:34:00.048541 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 12:34:00.050038 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:00.039061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46865 (local address 127.0.98.2:34897)
0619 12:34:00.040251 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:00.040308 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:00.042042 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:00.042673 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:00.042684 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:00.042803 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:00.043554 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:00.043577 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.044701 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.044704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:00.046201 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:00.046216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.046760 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.046766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:00.047003 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:00.047778 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:34:00.047782 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:00.047847 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:34:00.047947 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:00.048295 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:00.048307 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:00.048317 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 12:34:00.048473 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 12:34:00.048827 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":110,"threads_started":1}
I20260619 12:34:00.051373 19384 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46865
I20260619 12:34:00.051630 19384 heartbeater.cc:464] Registering TS with master...
I20260619 12:34:00.052174 19384 heartbeater.cc:511] Master 127.0.98.62:46865 requested a full tablet report, sending...
I20260619 12:34:00.053335 19061 ts_manager.cc:194] Registered new tserver with Master: ac3e8f2973fd423ba787b5220d4c6382 (127.0.98.2:41231)
I20260619 12:34:00.054243 19061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34897
W20260619 12:34:00.156795 19387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:00.157076 19387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:00.157191 19387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:00.161443 19387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:00.161502 19387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:00.161559 19387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:34:00.166353 19387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:00.167423 19387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:00.168874 19387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:00.178858 19395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:00.178858 19392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:00.179078 19387 server_base.cc:1034] running on GCE node
W20260619 12:34:00.178862 19393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:00.179750 19387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:00.180414 19387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:00.181610 19387 hybrid_clock.cc:648] HybridClock initialized: now 1781872440181594 us; error 30 us; skew 500 ppm
I20260619 12:34:00.183509 19387 webserver.cc:460] Webserver started at http://127.0.98.3:34633/ using document root <none> and password file <none>
I20260619 12:34:00.184099 19387 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:00.184194 19387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:00.184423 19387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:00.186525 19387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "925668262eae48398aa863c8d61c33ae"
format_stamp: "Formatted at 2026-06-19 12:34:00 on dist-test-slave-fg3h"
server_key: "2a0c481334a574a4d0dc06ae2bafa1f9"
server_key_iv: "4a1f58c85a7a226611ff6d416665ee0a"
server_key_version: "encryptionkey@0"
I20260619 12:34:00.187085 19387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "925668262eae48398aa863c8d61c33ae"
format_stamp: "Formatted at 2026-06-19 12:34:00 on dist-test-slave-fg3h"
server_key: "2a0c481334a574a4d0dc06ae2bafa1f9"
server_key_iv: "4a1f58c85a7a226611ff6d416665ee0a"
server_key_version: "encryptionkey@0"
I20260619 12:34:00.190806 19387 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 12:34:00.193722 19401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:00.195047 19387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:34:00.195189 19387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "925668262eae48398aa863c8d61c33ae"
format_stamp: "Formatted at 2026-06-19 12:34:00 on dist-test-slave-fg3h"
server_key: "2a0c481334a574a4d0dc06ae2bafa1f9"
server_key_iv: "4a1f58c85a7a226611ff6d416665ee0a"
server_key_version: "encryptionkey@0"
I20260619 12:34:00.195319 19387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:00.217072 19387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:00.217849 19387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:00.218053 19387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:00.218665 19387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:34:00.219645 19387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:34:00.219720 19387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:00.219795 19387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:34:00.219825 19387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:00.228343 19387 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41329
I20260619 12:34:00.228394 19514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41329 every 8 connection(s)
I20260619 12:34:00.229290 19387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:34:00.229955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 19387
I20260619 12:34:00.230074 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:34:00.230342 392 external_mini_cluster.cc:1428] Setting key 002662391e8f5e8efaf62c8401858bd3
I20260619 12:34:00.238569 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:00.231223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54419 (local address 127.0.98.62:46865)
0619 12:34:00.231373 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:00.231378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:00.232666 (+ 1288us) server_negotiation.cc:407] Connection header received
0619 12:34:00.234087 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:00.234090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:00.234135 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:00.234206 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:00.235113 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.235653 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.236756 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.236981 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.237061 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:00.238136 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:00.238139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:00.238142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:00.238153 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:00.238171 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:00.238178 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:00.238242 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:00.238403 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 12:34:00.238458 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:34:00.239421 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:00.231797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46865 (local address 127.0.98.3:54419)
0619 12:34:00.232527 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:00.232578 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:00.233897 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:00.234424 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:00.234431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:00.234515 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:00.234998 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:00.235009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.235766 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.235768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:00.236644 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:00.236653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.237164 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.237168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:00.237312 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:00.237898 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:34:00.237899 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:00.237936 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:34:00.237996 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:00.238279 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:00.238284 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:00.238289 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 12:34:00.238382 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 12:34:00.238592 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":141,"threads_started":1}
I20260619 12:34:00.240540 19515 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46865
I20260619 12:34:00.240816 19515 heartbeater.cc:464] Registering TS with master...
I20260619 12:34:00.241353 19515 heartbeater.cc:511] Master 127.0.98.62:46865 requested a full tablet report, sending...
I20260619 12:34:00.242336 19061 ts_manager.cc:194] Registered new tserver with Master: 925668262eae48398aa863c8d61c33ae (127.0.98.3:41329)
I20260619 12:34:00.242991 19061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54419
I20260619 12:34:00.244568 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:34:00.250764 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:00.246637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52698 (local address 127.0.98.62:46865)
0619 12:34:00.246849 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:00.246854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:00.246994 (+ 140us) server_negotiation.cc:407] Connection header received
0619 12:34:00.247141 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:00.247145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:00.247192 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:00.247317 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:00.247604 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.248108 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.249006 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.249315 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.249426 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:00.250256 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:00.250260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:00.250263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:00.250282 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:00.250300 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:00.250307 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:00.250435 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:00.250580 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 12:34:00.250620 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260619 12:34:00.253407 19061 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:34:00.256266 19061 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:34:00.270959 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:00.265042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41329 (local address 127.0.0.1:44628)
0619 12:34:00.265465 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:00.265525 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:00.265650 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:00.266382 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:00.266389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:00.266434 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:34:00.266572 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:00.266580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.267861 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.267864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:00.269216 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:00.269226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.270375 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.270378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:00.270496 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:00.270534 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 12:34:00.270660 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 12:34:00.270790 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":2,"thread_start_us":139,"threads_started":1}
I20260619 12:34:00.270983 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:00.265240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44628 (local address 127.0.98.3:41329)
0619 12:34:00.265632 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:00.265644 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:00.265669 (+ 25us) server_negotiation.cc:407] Connection header received
0619 12:34:00.265821 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:00.265830 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:00.266046 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:00.266203 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:34:00.266709 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.267751 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.269351 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.270029 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.270129 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:00.270313 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:00.270705 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 12:34:00.270815 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":99,"threads_started":1}
I20260619 12:34:00.271512 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:00.265406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36751 (local address 127.0.0.1:44654)
0619 12:34:00.266636 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:00.266649 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:00.266737 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:00.267250 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:00.267253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:00.267271 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:34:00.267354 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:00.267359 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.268638 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.268641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:00.269908 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:00.269918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.271130 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.271134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:00.271214 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:00.271231 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 12:34:00.271299 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 12:34:00.271378 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1162,"mutex_wait_us":25,"thread_start_us":121,"threads_started":1}
I20260619 12:34:00.271600 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:00.266033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44654 (local address 127.0.98.1:36751)
0619 12:34:00.266495 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:00.266506 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:00.266832 (+ 326us) server_negotiation.cc:407] Connection header received
0619 12:34:00.266913 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:00.266917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:00.267093 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:00.267247 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:34:00.267468 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.268554 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.270036 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.270794 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.270890 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:00.271058 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:00.271356 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 12:34:00.271452 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":143,"threads_started":1}
I20260619 12:34:00.272145 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:00.265895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41231 (local address 127.0.0.1:49156)
0619 12:34:00.266789 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:00.266802 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:00.266892 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:00.267403 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:00.267406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:00.267448 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:34:00.267534 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:00.267541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.268593 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.268599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:00.269983 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:00.269994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.270941 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.270944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:00.271847 (+ 903us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:00.271870 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:34:00.271952 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 12:34:00.272020 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":184,"thread_start_us":86,"threads_started":1}
I20260619 12:34:00.272264 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:00.266193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49156 (local address 127.0.98.2:41231)
0619 12:34:00.266637 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:00.266651 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:00.266876 (+ 225us) server_negotiation.cc:407] Connection header received
0619 12:34:00.266988 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:00.266993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:00.267132 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:00.267258 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:34:00.267661 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.268480 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.270079 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.270581 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.270705 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:00.270873 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:00.272001 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0619 12:34:00.272103 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":156,"threads_started":1}
I20260619 12:34:00.274123 19449 tablet_service.cc:1467] Processing CreateTablet for tablet e1dc5cb29e434318abf4dd4c0b05d14e (DEFAULT_TABLE table=test-table [id=9308a3888b734b5097237076bb1af647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:34:00.274147 19187 tablet_service.cc:1467] Processing CreateTablet for tablet e1dc5cb29e434318abf4dd4c0b05d14e (DEFAULT_TABLE table=test-table [id=9308a3888b734b5097237076bb1af647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:34:00.274984 19187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1dc5cb29e434318abf4dd4c0b05d14e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:00.274986 19449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1dc5cb29e434318abf4dd4c0b05d14e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:00.279587 19318 tablet_service.cc:1467] Processing CreateTablet for tablet e1dc5cb29e434318abf4dd4c0b05d14e (DEFAULT_TABLE table=test-table [id=9308a3888b734b5097237076bb1af647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:34:00.280668 19318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1dc5cb29e434318abf4dd4c0b05d14e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:00.281028 19532 tablet_bootstrap.cc:492] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae: Bootstrap starting.
I20260619 12:34:00.282925 19533 tablet_bootstrap.cc:492] T e1dc5cb29e434318abf4dd4c0b05d14e P a303b01a765e485c85eeb1d1659241a7: Bootstrap starting.
I20260619 12:34:00.283386 19532 tablet_bootstrap.cc:654] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:00.284102 19532 log.cc:824] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:00.285075 19533 tablet_bootstrap.cc:654] T e1dc5cb29e434318abf4dd4c0b05d14e P a303b01a765e485c85eeb1d1659241a7: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:00.285962 19532 tablet_bootstrap.cc:492] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae: No bootstrap required, opened a new log
I20260619 12:34:00.285930 19533 log.cc:824] T e1dc5cb29e434318abf4dd4c0b05d14e P a303b01a765e485c85eeb1d1659241a7: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:00.286185 19532 ts_tablet_manager.cc:1397] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 12:34:00.288020 19536 tablet_bootstrap.cc:492] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382: Bootstrap starting.
I20260619 12:34:00.290009 19536 tablet_bootstrap.cc:654] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:00.290207 19532 raft_consensus.cc:348] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925668262eae48398aa863c8d61c33ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41329 } } peers { permanent_uuid: "ac3e8f2973fd423ba787b5220d4c6382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41231 } } peers { permanent_uuid: "a303b01a765e485c85eeb1d1659241a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36751 } }
I20260619 12:34:00.290455 19532 raft_consensus.cc:374] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:00.290512 19532 raft_consensus.cc:729] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 925668262eae48398aa863c8d61c33ae, State: Initialized, Role: FOLLOWER
I20260619 12:34:00.290861 19536 log.cc:824] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:00.290956 19533 tablet_bootstrap.cc:492] T e1dc5cb29e434318abf4dd4c0b05d14e P a303b01a765e485c85eeb1d1659241a7: No bootstrap required, opened a new log
I20260619 12:34:00.291013 19532 consensus_queue.cc:260] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925668262eae48398aa863c8d61c33ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41329 } } peers { permanent_uuid: "ac3e8f2973fd423ba787b5220d4c6382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41231 } } peers { permanent_uuid: "a303b01a765e485c85eeb1d1659241a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36751 } }
I20260619 12:34:00.291142 19533 ts_tablet_manager.cc:1397] T e1dc5cb29e434318abf4dd4c0b05d14e P a303b01a765e485c85eeb1d1659241a7: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260619 12:34:00.291662 19515 heartbeater.cc:503] Master 127.0.98.62:46865 was elected leader, sending a full tablet report...
I20260619 12:34:00.292032 19532 ts_tablet_manager.cc:1428] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 12:34:00.292943 19536 tablet_bootstrap.cc:492] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382: No bootstrap required, opened a new log
I20260619 12:34:00.293140 19536 ts_tablet_manager.cc:1397] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 12:34:00.295469 19533 raft_consensus.cc:348] T e1dc5cb29e434318abf4dd4c0b05d14e P a303b01a765e485c85eeb1d1659241a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925668262eae48398aa863c8d61c33ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41329 } } peers { permanent_uuid: "ac3e8f2973fd423ba787b5220d4c6382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41231 } } peers { permanent_uuid: "a303b01a765e485c85eeb1d1659241a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36751 } }
I20260619 12:34:00.295739 19533 raft_consensus.cc:374] T e1dc5cb29e434318abf4dd4c0b05d14e P a303b01a765e485c85eeb1d1659241a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:00.295805 19533 raft_consensus.cc:729] T e1dc5cb29e434318abf4dd4c0b05d14e P a303b01a765e485c85eeb1d1659241a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a303b01a765e485c85eeb1d1659241a7, State: Initialized, Role: FOLLOWER
I20260619 12:34:00.296239 19533 consensus_queue.cc:260] T e1dc5cb29e434318abf4dd4c0b05d14e P a303b01a765e485c85eeb1d1659241a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925668262eae48398aa863c8d61c33ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41329 } } peers { permanent_uuid: "ac3e8f2973fd423ba787b5220d4c6382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41231 } } peers { permanent_uuid: "a303b01a765e485c85eeb1d1659241a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36751 } }
I20260619 12:34:00.296926 19253 heartbeater.cc:503] Master 127.0.98.62:46865 was elected leader, sending a full tablet report...
I20260619 12:34:00.297297 19533 ts_tablet_manager.cc:1428] T e1dc5cb29e434318abf4dd4c0b05d14e P a303b01a765e485c85eeb1d1659241a7: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260619 12:34:00.297259 19536 raft_consensus.cc:348] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925668262eae48398aa863c8d61c33ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41329 } } peers { permanent_uuid: "ac3e8f2973fd423ba787b5220d4c6382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41231 } } peers { permanent_uuid: "a303b01a765e485c85eeb1d1659241a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36751 } }
I20260619 12:34:00.297537 19536 raft_consensus.cc:374] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:00.297601 19536 raft_consensus.cc:729] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3e8f2973fd423ba787b5220d4c6382, State: Initialized, Role: FOLLOWER
I20260619 12:34:00.297999 19536 consensus_queue.cc:260] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925668262eae48398aa863c8d61c33ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41329 } } peers { permanent_uuid: "ac3e8f2973fd423ba787b5220d4c6382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41231 } } peers { permanent_uuid: "a303b01a765e485c85eeb1d1659241a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36751 } }
I20260619 12:34:00.298673 19384 heartbeater.cc:503] Master 127.0.98.62:46865 was elected leader, sending a full tablet report...
I20260619 12:34:00.298955 19536 ts_tablet_manager.cc:1428] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 12:34:00.304267 19540 raft_consensus.cc:482] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:34:00.304399 19540 raft_consensus.cc:504] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925668262eae48398aa863c8d61c33ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41329 } } peers { permanent_uuid: "ac3e8f2973fd423ba787b5220d4c6382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41231 } } peers { permanent_uuid: "a303b01a765e485c85eeb1d1659241a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36751 } }
I20260619 12:34:00.305459 19540 leader_election.cc:288] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 925668262eae48398aa863c8d61c33ae (127.0.98.3:41329), a303b01a765e485c85eeb1d1659241a7 (127.0.98.1:36751)
I20260619 12:34:00.309513 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:00.305662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36751 (local address 127.0.98.2:48223)
0619 12:34:00.305916 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:00.305932 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:00.306013 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:00.306323 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:00.306327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:00.306348 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:34:00.306452 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:00.306458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.307301 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.307304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:00.308412 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:00.308421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.309053 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.309057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:00.309182 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:00.309210 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 12:34:00.309314 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 12:34:00.309386 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":32}
I20260619 12:34:00.309511 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:00.305909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48223 (local address 127.0.98.1:36751)
0619 12:34:00.306014 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:00.306018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:00.306031 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:34:00.306121 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:00.306134 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:00.306178 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:00.306270 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:34:00.306563 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.307189 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.308533 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.308928 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.309041 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:00.309119 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:00.309331 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 12:34:00.309397 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 12:34:00.310185 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:00.305748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33837 (local address 127.0.98.3:41329)
0619 12:34:00.305900 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:00.305905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:00.306243 (+ 338us) server_negotiation.cc:407] Connection header received
0619 12:34:00.306411 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:00.306414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:00.306464 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:00.306558 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:34:00.306879 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.307695 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.309027 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.309499 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.309630 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:00.309711 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:00.310038 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 12:34:00.310100 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 12:34:00.310220 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:00.305682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41329 (local address 127.0.98.2:33837)
0619 12:34:00.306128 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:00.306143 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:00.306295 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:00.306566 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:00.306569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:00.306585 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:34:00.306728 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:00.306733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.307845 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.307853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:00.308832 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:00.308841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.309790 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.309793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:00.309897 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:00.309924 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 12:34:00.310011 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 12:34:00.310076 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":4,"thread_start_us":138,"threads_started":1}
I20260619 12:34:00.310201 19207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dc5cb29e434318abf4dd4c0b05d14e" candidate_uuid: "ac3e8f2973fd423ba787b5220d4c6382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a303b01a765e485c85eeb1d1659241a7" is_pre_election: true
I20260619 12:34:00.310501 19207 raft_consensus.cc:2454] T e1dc5cb29e434318abf4dd4c0b05d14e P a303b01a765e485c85eeb1d1659241a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3e8f2973fd423ba787b5220d4c6382 in term 0.
I20260619 12:34:00.311021 19271 leader_election.cc:302] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a303b01a765e485c85eeb1d1659241a7, ac3e8f2973fd423ba787b5220d4c6382; no voters:
I20260619 12:34:00.311046 19469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dc5cb29e434318abf4dd4c0b05d14e" candidate_uuid: "ac3e8f2973fd423ba787b5220d4c6382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "925668262eae48398aa863c8d61c33ae" is_pre_election: true
I20260619 12:34:00.311303 19469 raft_consensus.cc:2454] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3e8f2973fd423ba787b5220d4c6382 in term 0.
I20260619 12:34:00.311360 19540 raft_consensus.cc:2781] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:34:00.311448 19540 raft_consensus.cc:482] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:34:00.311504 19540 raft_consensus.cc:3037] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:00.312510 19540 raft_consensus.cc:504] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925668262eae48398aa863c8d61c33ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41329 } } peers { permanent_uuid: "ac3e8f2973fd423ba787b5220d4c6382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41231 } } peers { permanent_uuid: "a303b01a765e485c85eeb1d1659241a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36751 } }
I20260619 12:34:00.312906 19540 leader_election.cc:288] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [CANDIDATE]: Term 1 election: Requested vote from peers 925668262eae48398aa863c8d61c33ae (127.0.98.3:41329), a303b01a765e485c85eeb1d1659241a7 (127.0.98.1:36751)
I20260619 12:34:00.313274 19207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dc5cb29e434318abf4dd4c0b05d14e" candidate_uuid: "ac3e8f2973fd423ba787b5220d4c6382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a303b01a765e485c85eeb1d1659241a7"
I20260619 12:34:00.313408 19469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dc5cb29e434318abf4dd4c0b05d14e" candidate_uuid: "ac3e8f2973fd423ba787b5220d4c6382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "925668262eae48398aa863c8d61c33ae"
I20260619 12:34:00.313519 19207 raft_consensus.cc:3037] T e1dc5cb29e434318abf4dd4c0b05d14e P a303b01a765e485c85eeb1d1659241a7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:00.313553 19469 raft_consensus.cc:3037] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:00.314545 19469 raft_consensus.cc:2454] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3e8f2973fd423ba787b5220d4c6382 in term 1.
I20260619 12:34:00.314555 19207 raft_consensus.cc:2454] T e1dc5cb29e434318abf4dd4c0b05d14e P a303b01a765e485c85eeb1d1659241a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3e8f2973fd423ba787b5220d4c6382 in term 1.
I20260619 12:34:00.314857 19272 leader_election.cc:302] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 925668262eae48398aa863c8d61c33ae, ac3e8f2973fd423ba787b5220d4c6382; no voters:
I20260619 12:34:00.315006 19540 raft_consensus.cc:2781] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:00.315295 19540 raft_consensus.cc:686] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [term 1 LEADER]: Becoming Leader. State: Replica: ac3e8f2973fd423ba787b5220d4c6382, State: Running, Role: LEADER
I20260619 12:34:00.315637 19540 consensus_queue.cc:237] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925668262eae48398aa863c8d61c33ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41329 } } peers { permanent_uuid: "ac3e8f2973fd423ba787b5220d4c6382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41231 } } peers { permanent_uuid: "a303b01a765e485c85eeb1d1659241a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36751 } }
I20260619 12:34:00.318998 19061 catalog_manager.cc:5495] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac3e8f2973fd423ba787b5220d4c6382 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ac3e8f2973fd423ba787b5220d4c6382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "925668262eae48398aa863c8d61c33ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac3e8f2973fd423ba787b5220d4c6382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a303b01a765e485c85eeb1d1659241a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36751 } health_report { overall_health: UNKNOWN } } }
I20260619 12:34:00.336388 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:00.332723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49166 (local address 127.0.98.2:41231)
0619 12:34:00.332870 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:00.332874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:00.332889 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:34:00.333044 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:00.333046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:00.333150 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:00.333266 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:34:00.333520 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.334034 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.334904 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:00.335125 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:00.335235 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:00.335565 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:34:00.335963 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:34:00.336122 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:00.336209 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 12:34:00.336245 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:34:00.342269 19207 raft_consensus.cc:1261] T e1dc5cb29e434318abf4dd4c0b05d14e P a303b01a765e485c85eeb1d1659241a7 [term 1 FOLLOWER]: Refusing update from remote peer ac3e8f2973fd423ba787b5220d4c6382: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:34:00.342269 19469 raft_consensus.cc:1261] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae [term 1 FOLLOWER]: Refusing update from remote peer ac3e8f2973fd423ba787b5220d4c6382: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:34:00.343128 19542 consensus_queue.cc:1035] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925668262eae48398aa863c8d61c33ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:34:00.343417 19540 consensus_queue.cc:1035] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a303b01a765e485c85eeb1d1659241a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:34:00.352604 19547 mvcc.cc:205] Tried to move back new op lower bound from 7298549515634003968 to 7298549515535884288. Current Snapshot: MvccSnapshot[applied={T|T < 7298549515634003968}]
I20260619 12:34:00.353303 19546 mvcc.cc:205] Tried to move back new op lower bound from 7298549515634003968 to 7298549515535884288. Current Snapshot: MvccSnapshot[applied={T|T < 7298549515634003968}]
I20260619 12:34:00.360653 19061 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:34:00.360814 19061 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:34:00.363363 19061 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7f4530d0be347a6a0361edece5cf383: Sending DeleteTablet for 3 replicas of tablet e1dc5cb29e434318abf4dd4c0b05d14e
I20260619 12:34:00.364164 19449 tablet_service.cc:1514] Processing DeleteTablet for tablet e1dc5cb29e434318abf4dd4c0b05d14e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:34:00 UTC) from {username='slave'} at 127.0.0.1:44628
I20260619 12:34:00.364171 19318 tablet_service.cc:1514] Processing DeleteTablet for tablet e1dc5cb29e434318abf4dd4c0b05d14e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:34:00 UTC) from {username='slave'} at 127.0.0.1:49156
I20260619 12:34:00.364689 19554 tablet_replica.cc:330] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382: stopping tablet replica
I20260619 12:34:00.364665 19187 tablet_service.cc:1514] Processing DeleteTablet for tablet e1dc5cb29e434318abf4dd4c0b05d14e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:34:00 UTC) from {username='slave'} at 127.0.0.1:44654
I20260619 12:34:00.365064 19556 tablet_replica.cc:330] T e1dc5cb29e434318abf4dd4c0b05d14e P a303b01a765e485c85eeb1d1659241a7: stopping tablet replica
I20260619 12:34:00.365124 19554 raft_consensus.cc:2229] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:34:00.365391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 19125
I20260619 12:34:00.365391 19556 raft_consensus.cc:2229] T e1dc5cb29e434318abf4dd4c0b05d14e P a303b01a765e485c85eeb1d1659241a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:34:00.365660 19554 raft_consensus.cc:2258] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:34:00.367241 19554 ts_tablet_manager.cc:1905] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:34:00.369338 19555 tablet_replica.cc:330] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae: stopping tablet replica
I20260619 12:34:00.369670 19555 raft_consensus.cc:2229] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:34:00.370008 19555 raft_consensus.cc:2258] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:34:00.370847 19554 ts_tablet_manager.cc:1918] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:34:00.370994 19554 log.cc:1196] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/e1dc5cb29e434318abf4dd4c0b05d14e
I20260619 12:34:00.371407 19554 ts_tablet_manager.cc:1939] T e1dc5cb29e434318abf4dd4c0b05d14e P ac3e8f2973fd423ba787b5220d4c6382: Deleting consensus metadata
I20260619 12:34:00.371695 19555 ts_tablet_manager.cc:1905] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:34:00.372956 19050 catalog_manager.cc:4841] TS ac3e8f2973fd423ba787b5220d4c6382 (127.0.98.2:41231): tablet e1dc5cb29e434318abf4dd4c0b05d14e (table test-table [id=9308a3888b734b5097237076bb1af647]) successfully deleted
W20260619 12:34:00.373476 19049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36751 (error 108)
I20260619 12:34:00.373736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 19256
I20260619 12:34:00.375083 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:00.374750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36751 (local address 127.0.0.1:44670)
0619 12:34:00.374888 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:00.374963 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 12:34:00.375728 19049 catalog_manager.cc:4584] TS a303b01a765e485c85eeb1d1659241a7 (127.0.98.1:36751): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1dc5cb29e434318abf4dd4c0b05d14e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36751: connect: Connection refused (error 111)
I20260619 12:34:00.375910 19555 ts_tablet_manager.cc:1918] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:34:00.376012 19555 log.cc:1196] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/e1dc5cb29e434318abf4dd4c0b05d14e
I20260619 12:34:00.376286 19555 ts_tablet_manager.cc:1939] T e1dc5cb29e434318abf4dd4c0b05d14e P 925668262eae48398aa863c8d61c33ae: Deleting consensus metadata
I20260619 12:34:00.377466 19050 catalog_manager.cc:4841] TS 925668262eae48398aa863c8d61c33ae (127.0.98.3:41329): tablet e1dc5cb29e434318abf4dd4c0b05d14e (table test-table [id=9308a3888b734b5097237076bb1af647]) successfully deleted
I20260619 12:34:00.379707 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 19387
I20260619 12:34:00.385561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 19034
2026-06-19T12:34:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1008 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:00 dist-test-slave-fg3h krb5kdc[19560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:00 dist-test-slave-fg3h krb5kdc[19560](info): set up 2 sockets
Jun 19 12:34:00 dist-test-slave-fg3h krb5kdc[19560](info): commencing operation
krb5kdc: starting...
W20260619 12:34:01.425144 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:01 dist-test-slave-fg3h krb5kdc[19560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:34:01.548152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42757
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:01.660838 19573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:01.661188 19573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:01.661276 19573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:01.665256 19573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:01.665354 19573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:01.665391 19573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:01.665428 19573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:01.665463 19573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:01.670630 19573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:01.671669 19573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:01.672668 19573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:01.681015 19581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:01.681027 19579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:01.681238 19573 server_base.cc:1034] running on GCE node
W20260619 12:34:01.681051 19578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:01.682096 19573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:01.683202 19573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:01.684396 19573 hybrid_clock.cc:648] HybridClock initialized: now 1781872441684379 us; error 39 us; skew 500 ppm
Jun 19 12:34:01 dist-test-slave-fg3h krb5kdc[19560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:01.687141 19573 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:34:01.688026 19573 webserver.cc:460] Webserver started at http://127.0.98.62:42507/ using document root <none> and password file <none>
I20260619 12:34:01.688567 19573 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:01.688629 19573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:01.688791 19573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:01.690811 19573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "b57562d3669e4fdcaa09d30e95c7f50e"
format_stamp: "Formatted at 2026-06-19 12:34:01 on dist-test-slave-fg3h"
server_key: "c6cebad304dd3c53a1d7880f5994922a"
server_key_iv: "5148dde5e5ce34753194dea092496b86"
server_key_version: "encryptionkey@0"
I20260619 12:34:01.691315 19573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "b57562d3669e4fdcaa09d30e95c7f50e"
format_stamp: "Formatted at 2026-06-19 12:34:01 on dist-test-slave-fg3h"
server_key: "c6cebad304dd3c53a1d7880f5994922a"
server_key_iv: "5148dde5e5ce34753194dea092496b86"
server_key_version: "encryptionkey@0"
I20260619 12:34:01.694995 19573 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 12:34:01.697373 19588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:01.698568 19573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 12:34:01.698697 19573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "b57562d3669e4fdcaa09d30e95c7f50e"
format_stamp: "Formatted at 2026-06-19 12:34:01 on dist-test-slave-fg3h"
server_key: "c6cebad304dd3c53a1d7880f5994922a"
server_key_iv: "5148dde5e5ce34753194dea092496b86"
server_key_version: "encryptionkey@0"
I20260619 12:34:01.698815 19573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:01.715963 19573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:01.718957 19573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:01.719154 19573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:01.726208 19573 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42757
I20260619 12:34:01.726235 19640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42757 every 8 connection(s)
I20260619 12:34:01.727128 19573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:01.729794 19641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:01.734182 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 19573
I20260619 12:34:01.734328 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:01.734634 392 external_mini_cluster.cc:1428] Setting key ece490f92ef716798bfda22573beb800
I20260619 12:34:01.735543 19641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e: Bootstrap starting.
I20260619 12:34:01.737902 19641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:01.738679 19641 log.cc:824] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:01.740464 19641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e: No bootstrap required, opened a new log
Jun 19 12:34:01 dist-test-slave-fg3h krb5kdc[19560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:34:01.744577 19641 raft_consensus.cc:348] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57562d3669e4fdcaa09d30e95c7f50e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42757 } }
I20260619 12:34:01.744850 19641 raft_consensus.cc:374] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:01.744941 19641 raft_consensus.cc:729] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57562d3669e4fdcaa09d30e95c7f50e, State: Initialized, Role: FOLLOWER
I20260619 12:34:01.745476 19641 consensus_queue.cc:260] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57562d3669e4fdcaa09d30e95c7f50e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42757 } }
I20260619 12:34:01.745599 19641 raft_consensus.cc:388] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:01.745679 19641 raft_consensus.cc:482] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:01.745803 19641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:01.746681 19641 raft_consensus.cc:504] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57562d3669e4fdcaa09d30e95c7f50e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42757 } }
I20260619 12:34:01.747026 19641 leader_election.cc:302] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b57562d3669e4fdcaa09d30e95c7f50e; no voters:
I20260619 12:34:01.747364 19641 leader_election.cc:288] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:01.747450 19646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:01.747651 19646 raft_consensus.cc:686] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e [term 1 LEADER]: Becoming Leader. State: Replica: b57562d3669e4fdcaa09d30e95c7f50e, State: Running, Role: LEADER
I20260619 12:34:01.748010 19646 consensus_queue.cc:237] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57562d3669e4fdcaa09d30e95c7f50e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42757 } }
I20260619 12:34:01.748729 19641 sys_catalog.cc:564] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:01.749822 19648 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b57562d3669e4fdcaa09d30e95c7f50e. Latest consensus state: current_term: 1 leader_uuid: "b57562d3669e4fdcaa09d30e95c7f50e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57562d3669e4fdcaa09d30e95c7f50e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42757 } } }
I20260619 12:34:01.749972 19648 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:01.750084 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:01.736039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43882 (local address 127.0.98.62:42757)
0619 12:34:01.736443 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:01.736458 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:01.736491 (+ 33us) server_negotiation.cc:407] Connection header received
0619 12:34:01.737181 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:01.737199 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:01.737530 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:01.737857 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:01.738658 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:01.739656 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:01.740508 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:01.741030 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:01.741297 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:01.744103 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:01.744108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:01.744122 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:01.744150 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:01.746255 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:01.746704 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:01.746710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:01.746715 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:01.746883 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:01.747240 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:01.747246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:01.747250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:01.747789 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:01.748568 (+ 779us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:01.748717 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 12:34:01.749014 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":125,"threads_started":1}
I20260619 12:34:01.750357 19655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:01.749830 19647 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b57562d3669e4fdcaa09d30e95c7f50e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57562d3669e4fdcaa09d30e95c7f50e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42757 } } }
I20260619 12:34:01.750476 19647 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:01.753496 19655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:01.759168 19655 catalog_manager.cc:1347] Generated new cluster ID: 063ef4eed395412c9bacaf1f3c21a16c
I20260619 12:34:01.759233 19655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:01.767906 19655 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:01.769125 19655 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:01.781373 19655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e: Generated new TSK 0
I20260619 12:34:01.782194 19655 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 12:34:01.813244 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42757
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:01.927826 19667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:01.928165 19667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:01.928257 19667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:01.933735 19667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:01.933833 19667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:01.933951 19667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:34:01.939500 19667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:01.940595 19667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:01.941725 19667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:01.951629 19672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:01.951623 19673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:01.951615 19675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:01.952020 19667 server_base.cc:1034] running on GCE node
I20260619 12:34:01.952425 19667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:01.953063 19667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:01.954308 19667 hybrid_clock.cc:648] HybridClock initialized: now 1781872441954271 us; error 47 us; skew 500 ppm
Jun 19 12:34:01 dist-test-slave-fg3h krb5kdc[19560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:01.957224 19667 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 12:34:01.958127 19667 webserver.cc:460] Webserver started at http://127.0.98.1:45269/ using document root <none> and password file <none>
I20260619 12:34:01.958680 19667 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:01.958740 19667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:01.958956 19667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:01.961164 19667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "5b4278a3d7c04651b17d8160090fb755"
format_stamp: "Formatted at 2026-06-19 12:34:01 on dist-test-slave-fg3h"
server_key: "3f53037a033801c65f091821e7736abe"
server_key_iv: "2cd024e194173f6f3fc0288de901d281"
server_key_version: "encryptionkey@0"
I20260619 12:34:01.961736 19667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5b4278a3d7c04651b17d8160090fb755"
format_stamp: "Formatted at 2026-06-19 12:34:01 on dist-test-slave-fg3h"
server_key: "3f53037a033801c65f091821e7736abe"
server_key_iv: "2cd024e194173f6f3fc0288de901d281"
server_key_version: "encryptionkey@0"
I20260619 12:34:01.965519 19667 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:34:01.967911 19682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:01.969388 19667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:34:01.969527 19667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "5b4278a3d7c04651b17d8160090fb755"
format_stamp: "Formatted at 2026-06-19 12:34:01 on dist-test-slave-fg3h"
server_key: "3f53037a033801c65f091821e7736abe"
server_key_iv: "2cd024e194173f6f3fc0288de901d281"
server_key_version: "encryptionkey@0"
I20260619 12:34:01.969648 19667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:01.997268 19667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:02.000727 19667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:02.000958 19667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:02.001696 19667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:34:02.002681 19667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:34:02.002761 19667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:02.002832 19667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:34:02.002880 19667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:02.012573 19667 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42153
I20260619 12:34:02.012598 19795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42153 every 8 connection(s)
I20260619 12:34:02.013585 19667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:34:02.020365 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 19667
I20260619 12:34:02.020534 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:34:02.020879 392 external_mini_cluster.cc:1428] Setting key 1579295029122bec7523320bcd594094
Jun 19 12:34:02 dist-test-slave-fg3h krb5kdc[19560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:34:02.028502 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.015508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55837 (local address 127.0.98.62:42757)
0619 12:34:02.015698 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:02.015703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:02.016786 (+ 1083us) server_negotiation.cc:407] Connection header received
0619 12:34:02.017649 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:02.017652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:02.017698 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:02.017778 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:02.019000 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.019536 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.020807 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.021164 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.021293 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:02.024040 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:02.024043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:02.024045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:02.024068 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:02.025914 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:02.026605 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:02.026608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:02.026610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:02.026720 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:02.027150 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:02.027152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:02.027153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:02.027333 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:02.027411 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:02.028223 (+ 812us) server_negotiation.cc:299] Negotiation successful
0619 12:34:02.028371 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 12:34:02.029811 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.015930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42757 (local address 127.0.98.1:55837)
0619 12:34:02.016636 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:02.016677 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:02.017467 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:02.017950 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:02.017957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:02.018364 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:02.018851 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:02.018864 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.019647 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.019649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:02.020664 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:02.020681 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.021388 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.021394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:02.021619 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:02.022279 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:02.022295 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:02.023861 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:02.026051 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:02.026060 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:02.026080 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:02.026493 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:02.026828 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:02.026832 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:02.026834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:02.027048 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:02.027450 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:02.027459 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:02.027861 (+ 402us) client_negotiation.cc:769] Sending connection context
0619 12:34:02.028138 (+ 277us) client_negotiation.cc:240] Negotiation successful
0619 12:34:02.028549 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 12:34:02.031216 19796 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42757
I20260619 12:34:02.031471 19796 heartbeater.cc:464] Registering TS with master...
I20260619 12:34:02.031991 19796 heartbeater.cc:511] Master 127.0.98.62:42757 requested a full tablet report, sending...
I20260619 12:34:02.033736 19605 ts_manager.cc:194] Registered new tserver with Master: 5b4278a3d7c04651b17d8160090fb755 (127.0.98.1:42153)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 12:34:02.035087 19605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 12:34:02.052214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42757
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:02.166586 19801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:02.166888 19801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:02.166936 19801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:02.171095 19801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:02.171164 19801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:02.171226 19801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:34:02.176797 19801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:02.178025 19801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:02.179037 19801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:02.189121 19806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:02.189074 19807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:02.189100 19809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:02.189183 19801 server_base.cc:1034] running on GCE node
I20260619 12:34:02.189906 19801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:02.190508 19801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:02.191823 19801 hybrid_clock.cc:648] HybridClock initialized: now 1781872442191806 us; error 27 us; skew 500 ppm
Jun 19 12:34:02 dist-test-slave-fg3h krb5kdc[19560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:02.195000 19801 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 12:34:02.195951 19801 webserver.cc:460] Webserver started at http://127.0.98.2:44321/ using document root <none> and password file <none>
I20260619 12:34:02.196602 19801 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:02.196702 19801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:02.196949 19801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:02.199205 19801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "30fcb0f8698c4f3d90d10a3eb83ad0ab"
format_stamp: "Formatted at 2026-06-19 12:34:02 on dist-test-slave-fg3h"
server_key: "9f75060ed79f44e39ea4cc05b5e4e266"
server_key_iv: "26e9c95e8de55e7ca8a30e99bdbf7016"
server_key_version: "encryptionkey@0"
I20260619 12:34:02.199847 19801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "30fcb0f8698c4f3d90d10a3eb83ad0ab"
format_stamp: "Formatted at 2026-06-19 12:34:02 on dist-test-slave-fg3h"
server_key: "9f75060ed79f44e39ea4cc05b5e4e266"
server_key_iv: "26e9c95e8de55e7ca8a30e99bdbf7016"
server_key_version: "encryptionkey@0"
I20260619 12:34:02.204072 19801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 12:34:02.206902 19816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:02.208448 19801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:34:02.208601 19801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "30fcb0f8698c4f3d90d10a3eb83ad0ab"
format_stamp: "Formatted at 2026-06-19 12:34:02 on dist-test-slave-fg3h"
server_key: "9f75060ed79f44e39ea4cc05b5e4e266"
server_key_iv: "26e9c95e8de55e7ca8a30e99bdbf7016"
server_key_version: "encryptionkey@0"
I20260619 12:34:02.208741 19801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:02.226338 19801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:02.229977 19801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:02.230199 19801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:02.230870 19801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:34:02.231734 19801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:34:02.231805 19801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:02.231879 19801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:34:02.231922 19801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:02.240829 19801 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40389
I20260619 12:34:02.240906 19929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40389 every 8 connection(s)
I20260619 12:34:02.241807 19801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:34:02.249738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 19801
I20260619 12:34:02.249895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:34:02.250233 392 external_mini_cluster.cc:1428] Setting key b55f2c24fdb56ec9b48ee62f9fcec84c
Jun 19 12:34:02 dist-test-slave-fg3h krb5kdc[19560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:34:02.257364 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.243833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41283 (local address 127.0.98.62:42757)
0619 12:34:02.243995 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:02.243999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:02.244911 (+ 912us) server_negotiation.cc:407] Connection header received
0619 12:34:02.246366 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:02.246370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:02.246428 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:02.246540 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:02.247720 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.248261 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.249751 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.250114 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.250265 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:02.253058 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:02.253062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:02.253065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:02.253134 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:02.255006 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:02.255520 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:02.255526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:02.255531 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:02.255629 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:02.256041 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:02.256048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:02.256052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:02.256311 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:02.256421 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:02.256955 (+ 534us) server_negotiation.cc:299] Negotiation successful
0619 12:34:02.257125 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 12:34:02.258139 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.244257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42757 (local address 127.0.98.2:41283)
0619 12:34:02.244748 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:02.244791 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:02.246147 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:02.246719 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:02.246726 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:02.247083 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:02.247573 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:02.247586 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.248401 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.248405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:02.249568 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:02.249595 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.250339 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.250346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:02.250544 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:02.251260 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:02.251277 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:02.252892 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:02.255130 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:02.255136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:02.255153 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:02.255414 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:02.255736 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:02.255738 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:02.255740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:02.255916 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:02.256430 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:02.256435 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:02.256699 (+ 264us) client_negotiation.cc:769] Sending connection context
0619 12:34:02.256860 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 12:34:02.257202 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260619 12:34:02.259426 19930 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42757
I20260619 12:34:02.259735 19930 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 12:34:02.260342 19930 heartbeater.cc:511] Master 127.0.98.62:42757 requested a full tablet report, sending...
I20260619 12:34:02.261514 19605 ts_manager.cc:194] Registered new tserver with Master: 30fcb0f8698c4f3d90d10a3eb83ad0ab (127.0.98.2:40389)
I20260619 12:34:02.262128 19605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 12:34:02.280092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42757
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:02.392742 19935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:02.393013 19935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:02.393059 19935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:02.397058 19935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:02.397197 19935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:02.397262 19935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:34:02.402676 19935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:02.403738 19935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:02.404700 19935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:02.415337 19940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:02.415372 19941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:02.415378 19935 server_base.cc:1034] running on GCE node
W20260619 12:34:02.415337 19943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:02.416052 19935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:02.416680 19935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:02.417855 19935 hybrid_clock.cc:648] HybridClock initialized: now 1781872442417828 us; error 43 us; skew 500 ppm
Jun 19 12:34:02 dist-test-slave-fg3h krb5kdc[19560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:02.420651 19935 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 12:34:02.421657 19935 webserver.cc:460] Webserver started at http://127.0.98.3:36173/ using document root <none> and password file <none>
I20260619 12:34:02.422206 19935 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:02.422266 19935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:02.422434 19935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:02.424458 19935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "eb5f2ac93f4844baa9d717dab728714e"
format_stamp: "Formatted at 2026-06-19 12:34:02 on dist-test-slave-fg3h"
server_key: "31ca17b195849e960e612a9a9ac508d9"
server_key_iv: "46ef43769f4cf7516f0f2435c07e2f9a"
server_key_version: "encryptionkey@0"
I20260619 12:34:02.424991 19935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eb5f2ac93f4844baa9d717dab728714e"
format_stamp: "Formatted at 2026-06-19 12:34:02 on dist-test-slave-fg3h"
server_key: "31ca17b195849e960e612a9a9ac508d9"
server_key_iv: "46ef43769f4cf7516f0f2435c07e2f9a"
server_key_version: "encryptionkey@0"
I20260619 12:34:02.431015 19935 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.000s
I20260619 12:34:02.433811 19950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:02.435367 19935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:34:02.435523 19935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "eb5f2ac93f4844baa9d717dab728714e"
format_stamp: "Formatted at 2026-06-19 12:34:02 on dist-test-slave-fg3h"
server_key: "31ca17b195849e960e612a9a9ac508d9"
server_key_iv: "46ef43769f4cf7516f0f2435c07e2f9a"
server_key_version: "encryptionkey@0"
I20260619 12:34:02.435654 19935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:02.458396 19935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:02.461908 19935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:02.462137 19935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:02.462797 19935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:34:02.463726 19935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:34:02.463820 19935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:02.463899 19935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:34:02.463939 19935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:02.473489 19935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46589
I20260619 12:34:02.473556 20063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46589 every 8 connection(s)
I20260619 12:34:02.474611 19935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:34:02.477015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 19935
I20260619 12:34:02.477191 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:34:02.477587 392 external_mini_cluster.cc:1428] Setting key 1be03d9bbfaeb4bc244b00b0b0ef22f3
Jun 19 12:34:02 dist-test-slave-fg3h krb5kdc[19560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:34:02.489439 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.476650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51331 (local address 127.0.98.62:42757)
0619 12:34:02.476798 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:02.476802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:02.478147 (+ 1345us) server_negotiation.cc:407] Connection header received
0619 12:34:02.479562 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:02.479566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:02.479624 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:02.479702 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:02.480906 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.481644 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.482849 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.483083 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.483196 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:02.485839 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:02.485843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:02.485846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:02.485882 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:02.487279 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:02.487796 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:02.487801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:02.487803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:02.487898 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:02.488258 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:02.488261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:02.488263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:02.488454 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:02.488562 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:02.489144 (+ 582us) server_negotiation.cc:299] Negotiation successful
0619 12:34:02.489271 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:34:02.490283 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.477191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42757 (local address 127.0.98.3:51331)
0619 12:34:02.477782 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:02.477826 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:02.479349 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:02.479883 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:02.479890 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:02.480253 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:02.480757 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:02.480770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.481773 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.481777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:02.482697 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:02.482707 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.483244 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.483248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:02.483432 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:02.484091 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:02.484108 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:02.485656 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:02.487407 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:02.487413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:02.487429 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:02.487691 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:02.487996 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:02.487999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:02.488001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:02.488155 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:02.488558 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:02.488565 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:02.488859 (+ 294us) client_negotiation.cc:769] Sending connection context
0619 12:34:02.489069 (+ 210us) client_negotiation.cc:240] Negotiation successful
0619 12:34:02.489430 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":147,"threads_started":1}
I20260619 12:34:02.491514 20064 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42757
I20260619 12:34:02.491823 20064 heartbeater.cc:464] Registering TS with master...
I20260619 12:34:02.492434 20064 heartbeater.cc:511] Master 127.0.98.62:42757 requested a full tablet report, sending...
I20260619 12:34:02.493537 19605 ts_manager.cc:194] Registered new tserver with Master: eb5f2ac93f4844baa9d717dab728714e (127.0.98.3:46589)
I20260619 12:34:02.494103 19605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51331
I20260619 12:34:02.504014 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:34:02.513916 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.506112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.0.98.62:42757)
0619 12:34:02.506409 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:02.506414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:02.506552 (+ 138us) server_negotiation.cc:407] Connection header received
0619 12:34:02.506710 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:02.506713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:02.506762 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:02.506844 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:02.507601 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.508132 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.508980 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.509264 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.509372 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:02.510703 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:02.510707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:02.510709 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:02.510734 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:02.511948 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:02.512546 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:02.512550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:02.512551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:02.512645 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:02.512934 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:02.512937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:02.512938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:02.513230 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:02.513370 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:02.513621 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 12:34:02.513738 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260619 12:34:02.516727 19605 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:34:02.519722 19605 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:34:02.534888 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.529894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37288 (local address 127.0.98.3:46589)
0619 12:34:02.530393 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:02.530408 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:02.530435 (+ 27us) server_negotiation.cc:407] Connection header received
0619 12:34:02.530514 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:02.530519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:02.530719 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:02.530920 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:34:02.531136 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.532316 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.533552 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.534156 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.534270 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:02.534423 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:02.534610 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 12:34:02.534707 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":183,"threads_started":1}
I20260619 12:34:02.534901 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.529525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46589 (local address 127.0.0.1:37288)
0619 12:34:02.529773 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:02.529789 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:02.529893 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:02.530900 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:02.530903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:02.530922 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:34:02.531025 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:02.531032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.532444 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.532447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:02.533425 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:02.533435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.534277 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.534281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:02.534399 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:02.534442 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 12:34:02.534570 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 12:34:02.534701 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":1,"thread_start_us":105,"threads_started":1}
I20260619 12:34:02.535543 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.528719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42153 (local address 127.0.0.1:38034)
0619 12:34:02.529298 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:02.529353 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:02.529492 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:02.530109 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:02.530115 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:02.530140 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:34:02.530335 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:02.530345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.532001 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.532005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:02.533608 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:02.533619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.535047 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.535051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:02.535171 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:02.535195 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 12:34:02.535326 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 12:34:02.535406 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":31,"thread_start_us":100,"threads_started":1}
I20260619 12:34:02.535624 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.528952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38034 (local address 127.0.98.1:42153)
0619 12:34:02.529548 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:02.529568 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:02.529599 (+ 31us) server_negotiation.cc:407] Connection header received
0619 12:34:02.529685 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:02.529691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:02.529936 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:02.530100 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:34:02.530499 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.531856 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.533762 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.534717 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.534827 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:02.534975 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:02.535368 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 12:34:02.535473 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":146,"threads_started":1}
I20260619 12:34:02.536585 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.529846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40389 (local address 127.0.0.1:48038)
0619 12:34:02.530378 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:02.530394 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:02.530481 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:02.531157 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:02.531159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:02.531175 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:34:02.531250 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:02.531255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.532681 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.532684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:02.534019 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:02.534032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.536174 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.536178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:02.536280 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:02.536305 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 12:34:02.536386 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 12:34:02.536456 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":26,"thread_start_us":84,"threads_started":1}
I20260619 12:34:02.536660 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.530250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48038 (local address 127.0.98.2:40389)
0619 12:34:02.530705 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:02.530719 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:02.530742 (+ 23us) server_negotiation.cc:407] Connection header received
0619 12:34:02.530816 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:02.530820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:02.531013 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:02.531185 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:34:02.531345 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.532541 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.534156 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.536029 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.536166 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:02.536302 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:02.536424 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 12:34:02.536512 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":144,"threads_started":1}
I20260619 12:34:02.538133 19998 tablet_service.cc:1467] Processing CreateTablet for tablet 3b1c7b8ee652415186cc262e77c723f6 (DEFAULT_TABLE table=test-table [id=b822d864aaa343e2bfe50ce208c116a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:34:02.539350 19998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b1c7b8ee652415186cc262e77c723f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:02.539407 19864 tablet_service.cc:1467] Processing CreateTablet for tablet 3b1c7b8ee652415186cc262e77c723f6 (DEFAULT_TABLE table=test-table [id=b822d864aaa343e2bfe50ce208c116a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:34:02.539266 19730 tablet_service.cc:1467] Processing CreateTablet for tablet 3b1c7b8ee652415186cc262e77c723f6 (DEFAULT_TABLE table=test-table [id=b822d864aaa343e2bfe50ce208c116a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:34:02.540381 19864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b1c7b8ee652415186cc262e77c723f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:02.540381 19730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b1c7b8ee652415186cc262e77c723f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:02.545851 20081 tablet_bootstrap.cc:492] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e: Bootstrap starting.
I20260619 12:34:02.547828 20082 tablet_bootstrap.cc:492] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755: Bootstrap starting.
I20260619 12:34:02.548206 20081 tablet_bootstrap.cc:654] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:02.548277 20083 tablet_bootstrap.cc:492] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab: Bootstrap starting.
I20260619 12:34:02.549803 20081 log.cc:824] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:02.551017 20082 tablet_bootstrap.cc:654] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:02.551223 20083 tablet_bootstrap.cc:654] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:02.551784 20082 log.cc:824] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:02.552266 20081 tablet_bootstrap.cc:492] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e: No bootstrap required, opened a new log
I20260619 12:34:02.552505 20081 ts_tablet_manager.cc:1397] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 12:34:02.552484 20083 log.cc:824] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:02.553673 20082 tablet_bootstrap.cc:492] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755: No bootstrap required, opened a new log
I20260619 12:34:02.553906 20082 ts_tablet_manager.cc:1397] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260619 12:34:02.554566 20083 tablet_bootstrap.cc:492] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab: No bootstrap required, opened a new log
I20260619 12:34:02.554761 20083 ts_tablet_manager.cc:1397] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 12:34:02.557044 20081 raft_consensus.cc:348] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b4278a3d7c04651b17d8160090fb755" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42153 } } peers { permanent_uuid: "eb5f2ac93f4844baa9d717dab728714e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46589 } } peers { permanent_uuid: "30fcb0f8698c4f3d90d10a3eb83ad0ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } }
I20260619 12:34:02.557535 20081 raft_consensus.cc:374] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:02.557646 20081 raft_consensus.cc:729] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5f2ac93f4844baa9d717dab728714e, State: Initialized, Role: FOLLOWER
I20260619 12:34:02.557581 20082 raft_consensus.cc:348] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b4278a3d7c04651b17d8160090fb755" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42153 } } peers { permanent_uuid: "eb5f2ac93f4844baa9d717dab728714e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46589 } } peers { permanent_uuid: "30fcb0f8698c4f3d90d10a3eb83ad0ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } }
I20260619 12:34:02.557862 20082 raft_consensus.cc:374] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:02.557940 20082 raft_consensus.cc:729] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b4278a3d7c04651b17d8160090fb755, State: Initialized, Role: FOLLOWER
I20260619 12:34:02.558266 20081 consensus_queue.cc:260] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b4278a3d7c04651b17d8160090fb755" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42153 } } peers { permanent_uuid: "eb5f2ac93f4844baa9d717dab728714e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46589 } } peers { permanent_uuid: "30fcb0f8698c4f3d90d10a3eb83ad0ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } }
I20260619 12:34:02.558434 20082 consensus_queue.cc:260] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b4278a3d7c04651b17d8160090fb755" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42153 } } peers { permanent_uuid: "eb5f2ac93f4844baa9d717dab728714e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46589 } } peers { permanent_uuid: "30fcb0f8698c4f3d90d10a3eb83ad0ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } }
I20260619 12:34:02.559183 19796 heartbeater.cc:503] Master 127.0.98.62:42757 was elected leader, sending a full tablet report...
I20260619 12:34:02.559023 20083 raft_consensus.cc:348] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b4278a3d7c04651b17d8160090fb755" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42153 } } peers { permanent_uuid: "eb5f2ac93f4844baa9d717dab728714e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46589 } } peers { permanent_uuid: "30fcb0f8698c4f3d90d10a3eb83ad0ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } }
I20260619 12:34:02.559301 20083 raft_consensus.cc:374] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:02.559351 20083 raft_consensus.cc:729] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30fcb0f8698c4f3d90d10a3eb83ad0ab, State: Initialized, Role: FOLLOWER
I20260619 12:34:02.559408 20082 ts_tablet_manager.cc:1428] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260619 12:34:02.559772 20083 consensus_queue.cc:260] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b4278a3d7c04651b17d8160090fb755" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42153 } } peers { permanent_uuid: "eb5f2ac93f4844baa9d717dab728714e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46589 } } peers { permanent_uuid: "30fcb0f8698c4f3d90d10a3eb83ad0ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } }
I20260619 12:34:02.560060 20081 ts_tablet_manager.cc:1428] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260619 12:34:02.560434 20064 heartbeater.cc:503] Master 127.0.98.62:42757 was elected leader, sending a full tablet report...
I20260619 12:34:02.560436 19930 heartbeater.cc:503] Master 127.0.98.62:42757 was elected leader, sending a full tablet report...
I20260619 12:34:02.560714 20083 ts_tablet_manager.cc:1428] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 12:34:02.647038 20088 raft_consensus.cc:482] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:34:02.647294 20088 raft_consensus.cc:504] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b4278a3d7c04651b17d8160090fb755" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42153 } } peers { permanent_uuid: "eb5f2ac93f4844baa9d717dab728714e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46589 } } peers { permanent_uuid: "30fcb0f8698c4f3d90d10a3eb83ad0ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } }
I20260619 12:34:02.648497 20088 leader_election.cc:288] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb5f2ac93f4844baa9d717dab728714e (127.0.98.3:46589), 30fcb0f8698c4f3d90d10a3eb83ad0ab (127.0.98.2:40389)
I20260619 12:34:02.652896 20090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.648757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46589 (local address 127.0.98.1:34675)
0619 12:34:02.649159 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:02.649172 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:02.649366 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:02.649781 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:02.649784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:02.649802 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:34:02.649953 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:02.649960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.650792 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.650795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:02.651673 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:02.651683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.652383 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.652388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:02.652491 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:02.652559 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 12:34:02.652635 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 12:34:02.652722 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":28,"thread_start_us":167,"threads_started":1}
I20260619 12:34:02.652897 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.648877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34675 (local address 127.0.98.3:46589)
0619 12:34:02.649037 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:02.649043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:02.649342 (+ 299us) server_negotiation.cc:407] Connection header received
0619 12:34:02.649512 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:02.649515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:02.649576 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:02.649624 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:34:02.650091 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.650673 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.651812 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.652251 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.652336 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:02.652386 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:02.652673 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 12:34:02.652752 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 12:34:02.652936 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.649410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37285 (local address 127.0.98.2:40389)
0619 12:34:02.649535 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:02.649539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:02.649661 (+ 122us) server_negotiation.cc:407] Connection header received
0619 12:34:02.649781 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:02.649784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:02.649828 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:02.649923 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:34:02.650269 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.650805 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.651894 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.652340 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.652445 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:02.652519 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:02.652750 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 12:34:02.652822 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:34:02.652915 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.649276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40389 (local address 127.0.98.1:37285)
0619 12:34:02.649574 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:02.649586 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:02.649675 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:02.649933 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:02.649935 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:02.649957 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:34:02.650155 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:02.650162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.650946 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.650949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:02.651764 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:02.651776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.652483 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.652486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:02.652590 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:02.652629 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 12:34:02.652707 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 12:34:02.652772 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260619 12:34:02.653836 19884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b1c7b8ee652415186cc262e77c723f6" candidate_uuid: "5b4278a3d7c04651b17d8160090fb755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30fcb0f8698c4f3d90d10a3eb83ad0ab" is_pre_election: true
I20260619 12:34:02.653858 20018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b1c7b8ee652415186cc262e77c723f6" candidate_uuid: "5b4278a3d7c04651b17d8160090fb755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb5f2ac93f4844baa9d717dab728714e" is_pre_election: true
I20260619 12:34:02.654142 19884 raft_consensus.cc:2454] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b4278a3d7c04651b17d8160090fb755 in term 0.
I20260619 12:34:02.654258 20018 raft_consensus.cc:2454] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b4278a3d7c04651b17d8160090fb755 in term 0.
I20260619 12:34:02.654646 19686 leader_election.cc:302] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30fcb0f8698c4f3d90d10a3eb83ad0ab, 5b4278a3d7c04651b17d8160090fb755; no voters:
I20260619 12:34:02.654934 20088 raft_consensus.cc:2781] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:34:02.655032 20088 raft_consensus.cc:482] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:34:02.655095 20088 raft_consensus.cc:3037] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:02.656153 20088 raft_consensus.cc:504] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b4278a3d7c04651b17d8160090fb755" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42153 } } peers { permanent_uuid: "eb5f2ac93f4844baa9d717dab728714e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46589 } } peers { permanent_uuid: "30fcb0f8698c4f3d90d10a3eb83ad0ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } }
I20260619 12:34:02.656504 20088 leader_election.cc:288] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 [CANDIDATE]: Term 1 election: Requested vote from peers eb5f2ac93f4844baa9d717dab728714e (127.0.98.3:46589), 30fcb0f8698c4f3d90d10a3eb83ad0ab (127.0.98.2:40389)
I20260619 12:34:02.656927 20018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b1c7b8ee652415186cc262e77c723f6" candidate_uuid: "5b4278a3d7c04651b17d8160090fb755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb5f2ac93f4844baa9d717dab728714e"
I20260619 12:34:02.656968 19884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b1c7b8ee652415186cc262e77c723f6" candidate_uuid: "5b4278a3d7c04651b17d8160090fb755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30fcb0f8698c4f3d90d10a3eb83ad0ab"
I20260619 12:34:02.657068 19884 raft_consensus.cc:3037] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:02.657066 20018 raft_consensus.cc:3037] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:02.658393 19884 raft_consensus.cc:2454] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b4278a3d7c04651b17d8160090fb755 in term 1.
I20260619 12:34:02.658394 20018 raft_consensus.cc:2454] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b4278a3d7c04651b17d8160090fb755 in term 1.
I20260619 12:34:02.658682 19686 leader_election.cc:302] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b4278a3d7c04651b17d8160090fb755, eb5f2ac93f4844baa9d717dab728714e; no voters:
I20260619 12:34:02.658913 20088 raft_consensus.cc:2781] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:02.659161 20088 raft_consensus.cc:686] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 [term 1 LEADER]: Becoming Leader. State: Replica: 5b4278a3d7c04651b17d8160090fb755, State: Running, Role: LEADER
I20260619 12:34:02.659544 20088 consensus_queue.cc:237] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b4278a3d7c04651b17d8160090fb755" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42153 } } peers { permanent_uuid: "eb5f2ac93f4844baa9d717dab728714e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46589 } } peers { permanent_uuid: "30fcb0f8698c4f3d90d10a3eb83ad0ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } }
I20260619 12:34:02.662750 19605 catalog_manager.cc:5495] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b4278a3d7c04651b17d8160090fb755 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5b4278a3d7c04651b17d8160090fb755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b4278a3d7c04651b17d8160090fb755" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5f2ac93f4844baa9d717dab728714e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30fcb0f8698c4f3d90d10a3eb83ad0ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 } health_report { overall_health: UNKNOWN } } }
I20260619 12:34:02.685922 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.682523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38038 (local address 127.0.98.1:42153)
0619 12:34:02.682681 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:02.682685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:02.682698 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:34:02.682772 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:02.682774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:02.682827 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:02.682919 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:34:02.683195 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.683709 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.684553 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.684768 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.684855 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:02.685142 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:34:02.685592 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:34:02.685686 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:02.685766 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 12:34:02.685816 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 12:34:02.692055 20018 raft_consensus.cc:1261] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e [term 1 FOLLOWER]: Refusing update from remote peer 5b4278a3d7c04651b17d8160090fb755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:34:02.692055 19884 raft_consensus.cc:1261] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab [term 1 FOLLOWER]: Refusing update from remote peer 5b4278a3d7c04651b17d8160090fb755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:34:02.692762 20092 consensus_queue.cc:1035] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5f2ac93f4844baa9d717dab728714e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:34:02.692970 20088 consensus_queue.cc:1035] T 3b1c7b8ee652415186cc262e77c723f6 P 5b4278a3d7c04651b17d8160090fb755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30fcb0f8698c4f3d90d10a3eb83ad0ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:34:02.700469 20095 mvcc.cc:205] Tried to move back new op lower bound from 7298549525258657792 to 7298549525136109568. Current Snapshot: MvccSnapshot[applied={T|T < 7298549525258657792}]
I20260619 12:34:02.709697 19605 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:34:02.709884 19605 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:34:02.712275 19605 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b57562d3669e4fdcaa09d30e95c7f50e: Sending DeleteTablet for 3 replicas of tablet 3b1c7b8ee652415186cc262e77c723f6
I20260619 12:34:02.713253 19998 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b1c7b8ee652415186cc262e77c723f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:34:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37288
I20260619 12:34:02.713513 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 19667
I20260619 12:34:02.714509 20105 tablet_replica.cc:330] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e: stopping tablet replica
I20260619 12:34:02.714597 19864 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b1c7b8ee652415186cc262e77c723f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:34:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48038
I20260619 12:34:02.714919 20105 raft_consensus.cc:2229] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:34:02.715277 20105 raft_consensus.cc:2258] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:34:02.715893 20106 tablet_replica.cc:330] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab: stopping tablet replica
I20260619 12:34:02.716217 20106 raft_consensus.cc:2229] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:34:02.716519 20106 raft_consensus.cc:2258] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:34:02.717028 20105 ts_tablet_manager.cc:1905] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:34:02.718029 20106 ts_tablet_manager.cc:1905] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:34:02.719266 20105 ts_tablet_manager.cc:1918] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:34:02.719374 20105 log.cc:1196] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/3b1c7b8ee652415186cc262e77c723f6
I20260619 12:34:02.719726 20105 ts_tablet_manager.cc:1939] T 3b1c7b8ee652415186cc262e77c723f6 P eb5f2ac93f4844baa9d717dab728714e: Deleting consensus metadata
I20260619 12:34:02.720845 19592 catalog_manager.cc:4841] TS eb5f2ac93f4844baa9d717dab728714e (127.0.98.3:46589): tablet 3b1c7b8ee652415186cc262e77c723f6 (table test-table [id=b822d864aaa343e2bfe50ce208c116a9]) successfully deleted
W20260619 12:34:02.721418 19592 connection.cc:535] client connection to 127.0.98.1:42153 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 12:34:02.721448 20106 ts_tablet_manager.cc:1918] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:34:02.721537 20106 log.cc:1196] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/3b1c7b8ee652415186cc262e77c723f6
W20260619 12:34:02.721565 19592 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 12:34:02.721732 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 19801
I20260619 12:34:02.721865 20106 ts_tablet_manager.cc:1939] T 3b1c7b8ee652415186cc262e77c723f6 P 30fcb0f8698c4f3d90d10a3eb83ad0ab: Deleting consensus metadata
I20260619 12:34:02.723654 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.723332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42153 (local address 127.0.0.1:38044)
0619 12:34:02.723513 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:02.723582 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260619 12:34:02.723801 19592 catalog_manager.cc:4584] TS 5b4278a3d7c04651b17d8160090fb755 (127.0.98.1:42153): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b1c7b8ee652415186cc262e77c723f6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42153: connect: Connection refused (error 111)
I20260619 12:34:02.727690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 19935
I20260619 12:34:02.729920 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.729645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40389 (local address 127.0.0.1:48044)
0619 12:34:02.729788 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:02.729847 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 12:34:02.734189 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 19573
2026-06-19T12:34:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2349 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T12:34:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:02Z Disabled control of system clock
I20260619 12:34:02.764683 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46307
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 12:34:02.879845 20111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:02.880122 20111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:02.880170 20111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:02.884285 20111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:02.884341 20111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:02.884357 20111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:02.884374 20111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:02.884389 20111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:02.889142 20111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46307
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:02.890241 20111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:02.891579 20111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:02.899925 20116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:02.899968 20117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:02.900162 20111 server_base.cc:1034] running on GCE node
W20260619 12:34:02.899945 20119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:02.900844 20111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:02.902037 20111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:02.903246 20111 hybrid_clock.cc:648] HybridClock initialized: now 1781872442903214 us; error 51 us; skew 500 ppm
I20260619 12:34:02.905381 20111 webserver.cc:460] Webserver started at http://127.0.98.62:36881/ using document root <none> and password file <none>
I20260619 12:34:02.905985 20111 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:02.906083 20111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:02.906330 20111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:02.908444 20111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "a173c93cbf1049ee804412dc3e543a37"
format_stamp: "Formatted at 2026-06-19 12:34:02 on dist-test-slave-fg3h"
server_key: "7d38e38ef50c898eb25cbb90e35b6bdc"
server_key_iv: "2d3309beba5e0eccddb3d886d68ae3ea"
server_key_version: "encryptionkey@0"
I20260619 12:34:02.909029 20111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "a173c93cbf1049ee804412dc3e543a37"
format_stamp: "Formatted at 2026-06-19 12:34:02 on dist-test-slave-fg3h"
server_key: "7d38e38ef50c898eb25cbb90e35b6bdc"
server_key_iv: "2d3309beba5e0eccddb3d886d68ae3ea"
server_key_version: "encryptionkey@0"
I20260619 12:34:02.912817 20111 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:34:02.915411 20125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:02.916659 20111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 12:34:02.916801 20111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "a173c93cbf1049ee804412dc3e543a37"
format_stamp: "Formatted at 2026-06-19 12:34:02 on dist-test-slave-fg3h"
server_key: "7d38e38ef50c898eb25cbb90e35b6bdc"
server_key_iv: "2d3309beba5e0eccddb3d886d68ae3ea"
server_key_version: "encryptionkey@0"
I20260619 12:34:02.916917 20111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:02.937314 20111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:02.938059 20111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:02.938210 20111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:02.945137 20111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46307
I20260619 12:34:02.945143 20177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46307 every 8 connection(s)
I20260619 12:34:02.946095 20111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:02.949077 20178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:02.950807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 20111
I20260619 12:34:02.950941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:02.951216 392 external_mini_cluster.cc:1428] Setting key 5712c9a4df26a3a4987691bac97141f6
I20260619 12:34:02.955106 20178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37: Bootstrap starting.
I20260619 12:34:02.957518 20178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:02.958268 20178 log.cc:824] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:02.960423 20178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37: No bootstrap required, opened a new log
I20260619 12:34:02.960886 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:02.952639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47888 (local address 127.0.98.62:46307)
0619 12:34:02.953306 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:02.953321 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:02.953357 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:34:02.954196 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:02.954216 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:02.954639 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:02.954913 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:02.955311 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.956241 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.957427 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:02.957961 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:02.958081 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:02.959082 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:02.959090 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:02.959109 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:02.959507 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:02.959561 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:02.959581 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:02.959660 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:02.959857 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 12:34:02.960147 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":134,"threads_started":1}
I20260619 12:34:02.964347 20178 raft_consensus.cc:348] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a173c93cbf1049ee804412dc3e543a37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } }
I20260619 12:34:02.964556 20178 raft_consensus.cc:374] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:02.964628 20178 raft_consensus.cc:729] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a173c93cbf1049ee804412dc3e543a37, State: Initialized, Role: FOLLOWER
I20260619 12:34:02.965035 20178 consensus_queue.cc:260] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a173c93cbf1049ee804412dc3e543a37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } }
I20260619 12:34:02.965233 20178 raft_consensus.cc:388] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:02.965299 20178 raft_consensus.cc:482] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:02.965400 20178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:02.966274 20178 raft_consensus.cc:504] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a173c93cbf1049ee804412dc3e543a37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } }
I20260619 12:34:02.966603 20178 leader_election.cc:302] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a173c93cbf1049ee804412dc3e543a37; no voters:
I20260619 12:34:02.966920 20178 leader_election.cc:288] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:02.967092 20183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:02.967475 20183 raft_consensus.cc:686] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37 [term 1 LEADER]: Becoming Leader. State: Replica: a173c93cbf1049ee804412dc3e543a37, State: Running, Role: LEADER
I20260619 12:34:02.967803 20183 consensus_queue.cc:237] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a173c93cbf1049ee804412dc3e543a37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } }
I20260619 12:34:02.968146 20178 sys_catalog.cc:564] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:02.969419 20185 sys_catalog.cc:455] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a173c93cbf1049ee804412dc3e543a37. Latest consensus state: current_term: 1 leader_uuid: "a173c93cbf1049ee804412dc3e543a37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a173c93cbf1049ee804412dc3e543a37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } } }
I20260619 12:34:02.969553 20185 sys_catalog.cc:458] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:02.969929 20184 sys_catalog.cc:455] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a173c93cbf1049ee804412dc3e543a37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a173c93cbf1049ee804412dc3e543a37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46307 } } }
I20260619 12:34:02.970028 20184 sys_catalog.cc:458] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:02.969987 20192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:02.972980 20192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:02.978698 20192 catalog_manager.cc:1347] Generated new cluster ID: dc58037709574753953bc9a67698953b
I20260619 12:34:02.978777 20192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:02.985955 20192 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:02.987175 20192 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:02.992455 20192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37: Generated new TSK 0
I20260619 12:34:02.993207 20192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:34:03.005125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46307
--builtin_ntp_servers=127.0.98.20:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 12:34:03.120442 20202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:03.120704 20202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:03.120771 20202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:03.124934 20202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:03.125001 20202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:03.125094 20202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 12:34:03.129976 20202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:03.131026 20202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:03.132419 20202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:03.141805 20208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:03.141795 20207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:03.141811 20210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:03.142143 20202 server_base.cc:1034] running on GCE node
I20260619 12:34:03.142555 20202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:03.143172 20202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:03.144369 20202 hybrid_clock.cc:648] HybridClock initialized: now 1781872443144333 us; error 48 us; skew 500 ppm
I20260619 12:34:03.146246 20202 webserver.cc:460] Webserver started at http://127.0.98.1:43103/ using document root <none> and password file <none>
I20260619 12:34:03.146858 20202 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:03.146955 20202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:03.147187 20202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:03.149297 20202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-0/data/instance:
uuid: "4163003a226d42b6b7b49b210e771e2b"
format_stamp: "Formatted at 2026-06-19 12:34:03 on dist-test-slave-fg3h"
server_key: "80682d7029e8d5ae1939da258eff31d0"
server_key_iv: "97b39e798784799fe87b4a5c39cbbd5c"
server_key_version: "encryptionkey@0"
I20260619 12:34:03.149860 20202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4163003a226d42b6b7b49b210e771e2b"
format_stamp: "Formatted at 2026-06-19 12:34:03 on dist-test-slave-fg3h"
server_key: "80682d7029e8d5ae1939da258eff31d0"
server_key_iv: "97b39e798784799fe87b4a5c39cbbd5c"
server_key_version: "encryptionkey@0"
I20260619 12:34:03.153551 20202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:34:03.155966 20216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:03.157250 20202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:34:03.157382 20202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-0/wal
uuid: "4163003a226d42b6b7b49b210e771e2b"
format_stamp: "Formatted at 2026-06-19 12:34:03 on dist-test-slave-fg3h"
server_key: "80682d7029e8d5ae1939da258eff31d0"
server_key_iv: "97b39e798784799fe87b4a5c39cbbd5c"
server_key_version: "encryptionkey@0"
I20260619 12:34:03.157500 20202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:03.176158 20202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:03.176865 20202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:03.177074 20202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:03.177744 20202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:34:03.178690 20202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:34:03.178763 20202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:03.178833 20202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:34:03.178876 20202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:03.187685 20202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44107
I20260619 12:34:03.187717 20329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44107 every 8 connection(s)
I20260619 12:34:03.188594 20202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-0/data/info.pb
I20260619 12:34:03.192103 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 20202
I20260619 12:34:03.192245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-0/wal/instance
I20260619 12:34:03.192538 392 external_mini_cluster.cc:1428] Setting key aa42075a03c2ff843313f00fa4d51bfa
I20260619 12:34:03.194393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46307
--builtin_ntp_servers=127.0.98.20:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 12:34:03.198268 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:03.190549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51315 (local address 127.0.98.62:46307)
0619 12:34:03.190696 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:03.190699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:03.191788 (+ 1089us) server_negotiation.cc:407] Connection header received
0619 12:34:03.193348 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:03.193352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:03.193407 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:03.193517 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:03.194404 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.195017 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.196157 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.196393 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.196503 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:03.197665 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:03.197669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:03.197671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:03.197684 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:03.197707 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:03.197719 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:03.197803 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:03.198054 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 12:34:03.198155 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 12:34:03.199479 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:03.190955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.1:51315)
0619 12:34:03.191631 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:03.191672 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:03.193129 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:03.193688 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:03.193695 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:03.193785 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:03.194260 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:03.194272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.195147 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.195150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:03.196026 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:03.196035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.196552 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.196557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:03.196716 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:03.197411 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:34:03.197413 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:03.197451 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:34:03.197514 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:03.197838 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:03.197850 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:03.197859 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 12:34:03.198013 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 12:34:03.198352 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":116,"threads_started":1}
I20260619 12:34:03.200800 20330 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46307
I20260619 12:34:03.201033 20330 heartbeater.cc:464] Registering TS with master...
I20260619 12:34:03.201562 20330 heartbeater.cc:511] Master 127.0.98.62:46307 requested a full tablet report, sending...
I20260619 12:34:03.203065 20142 ts_manager.cc:194] Registered new tserver with Master: 4163003a226d42b6b7b49b210e771e2b (127.0.98.1:44107)
I20260619 12:34:03.204175 20142 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51315
W20260619 12:34:03.304106 20333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:03.304427 20333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:03.304502 20333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:03.308640 20333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:03.308733 20333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:03.308831 20333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 12:34:03.313551 20333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:03.314729 20333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:03.316134 20333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:03.325950 20338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:03.326056 20333 server_base.cc:1034] running on GCE node
W20260619 12:34:03.325987 20339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:03.325949 20341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:03.326632 20333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:03.327297 20333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:03.328493 20333 hybrid_clock.cc:648] HybridClock initialized: now 1781872443328476 us; error 29 us; skew 500 ppm
I20260619 12:34:03.330526 20333 webserver.cc:460] Webserver started at http://127.0.98.2:42227/ using document root <none> and password file <none>
I20260619 12:34:03.331128 20333 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:03.331198 20333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:03.331424 20333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:03.333560 20333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-1/data/instance:
uuid: "cbbfa688bdc1413a93e15114c2a334fe"
format_stamp: "Formatted at 2026-06-19 12:34:03 on dist-test-slave-fg3h"
server_key: "4bb63681d5573a77a992c99e95e16e21"
server_key_iv: "6b473afa50197bd0aa68b29f64838a76"
server_key_version: "encryptionkey@0"
I20260619 12:34:03.334151 20333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cbbfa688bdc1413a93e15114c2a334fe"
format_stamp: "Formatted at 2026-06-19 12:34:03 on dist-test-slave-fg3h"
server_key: "4bb63681d5573a77a992c99e95e16e21"
server_key_iv: "6b473afa50197bd0aa68b29f64838a76"
server_key_version: "encryptionkey@0"
I20260619 12:34:03.337900 20333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 12:34:03.340307 20347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:03.341648 20333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:34:03.341796 20333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-1/wal
uuid: "cbbfa688bdc1413a93e15114c2a334fe"
format_stamp: "Formatted at 2026-06-19 12:34:03 on dist-test-slave-fg3h"
server_key: "4bb63681d5573a77a992c99e95e16e21"
server_key_iv: "6b473afa50197bd0aa68b29f64838a76"
server_key_version: "encryptionkey@0"
I20260619 12:34:03.341908 20333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:03.381011 20333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:03.381832 20333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:03.382047 20333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:03.382679 20333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:34:03.383644 20333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:34:03.383718 20333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:03.383790 20333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:34:03.383841 20333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:03.392966 20333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36829
I20260619 12:34:03.392985 20460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36829 every 8 connection(s)
I20260619 12:34:03.393922 20333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-1/data/info.pb
I20260619 12:34:03.401367 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 20333
I20260619 12:34:03.401516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-1/wal/instance
I20260619 12:34:03.401852 392 external_mini_cluster.cc:1428] Setting key 619c1cabff7d105d83b8e3b4bfcb440b
I20260619 12:34:03.404315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46307
--builtin_ntp_servers=127.0.98.20:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 12:34:03.404551 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:03.395847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52397 (local address 127.0.98.62:46307)
0619 12:34:03.395995 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:03.395999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:03.397120 (+ 1121us) server_negotiation.cc:407] Connection header received
0619 12:34:03.398701 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:03.398705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:03.398766 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:03.398903 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:03.399879 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.400473 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.402181 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.402514 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.402614 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:03.403994 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:03.403997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:03.403999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:03.404011 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:03.404032 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:03.404040 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:03.404095 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:03.404303 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 12:34:03.404356 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 12:34:03.405956 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:03.396253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.2:52397)
0619 12:34:03.396939 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:03.396983 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:03.398462 (+ 1479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:03.399093 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:03.399101 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:03.399195 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:03.399714 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:03.399727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.400631 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.400634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:03.402049 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:03.402062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.402683 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.402687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:03.402911 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:03.403669 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:34:03.403671 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:03.403730 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:34:03.403818 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:03.404128 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:03.404142 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:03.404152 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 12:34:03.404304 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 12:34:03.404664 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":113,"threads_started":1}
I20260619 12:34:03.407358 20461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46307
I20260619 12:34:03.407634 20461 heartbeater.cc:464] Registering TS with master...
I20260619 12:34:03.408221 20461 heartbeater.cc:511] Master 127.0.98.62:46307 requested a full tablet report, sending...
I20260619 12:34:03.409511 20142 ts_manager.cc:194] Registered new tserver with Master: cbbfa688bdc1413a93e15114c2a334fe (127.0.98.2:36829)
I20260619 12:34:03.410418 20142 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:52397
W20260619 12:34:03.517935 20464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:03.518215 20464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:03.518255 20464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:03.522469 20464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:03.522531 20464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:03.522590 20464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 12:34:03.527415 20464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:03.528522 20464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:03.529892 20464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:03.539746 20469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:03.539754 20472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:03.539997 20464 server_base.cc:1034] running on GCE node
W20260619 12:34:03.540114 20470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:03.540524 20464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:03.541265 20464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:03.542454 20464 hybrid_clock.cc:648] HybridClock initialized: now 1781872443542417 us; error 48 us; skew 500 ppm
I20260619 12:34:03.544549 20464 webserver.cc:460] Webserver started at http://127.0.98.3:44547/ using document root <none> and password file <none>
I20260619 12:34:03.545274 20464 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:03.545346 20464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:03.545616 20464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:03.547722 20464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-2/data/instance:
uuid: "339aaf3bc0654d94b7e89b3ef4614c04"
format_stamp: "Formatted at 2026-06-19 12:34:03 on dist-test-slave-fg3h"
server_key: "7d5c8157261af9d631ff5ff1f9ef915f"
server_key_iv: "bba71979b33b23e9f0a5b582787d5124"
server_key_version: "encryptionkey@0"
I20260619 12:34:03.548309 20464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-2/wal/instance:
uuid: "339aaf3bc0654d94b7e89b3ef4614c04"
format_stamp: "Formatted at 2026-06-19 12:34:03 on dist-test-slave-fg3h"
server_key: "7d5c8157261af9d631ff5ff1f9ef915f"
server_key_iv: "bba71979b33b23e9f0a5b582787d5124"
server_key_version: "encryptionkey@0"
I20260619 12:34:03.552124 20464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:34:03.554617 20478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:03.555871 20464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:34:03.555985 20464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-2/wal
uuid: "339aaf3bc0654d94b7e89b3ef4614c04"
format_stamp: "Formatted at 2026-06-19 12:34:03 on dist-test-slave-fg3h"
server_key: "7d5c8157261af9d631ff5ff1f9ef915f"
server_key_iv: "bba71979b33b23e9f0a5b582787d5124"
server_key_version: "encryptionkey@0"
I20260619 12:34:03.556092 20464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:03.577397 20464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:03.578079 20464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:03.578285 20464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:03.578879 20464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 12:34:03.579808 20464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 12:34:03.579862 20464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:03.579929 20464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 12:34:03.579968 20464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:03.588917 20464 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34845
I20260619 12:34:03.588939 20591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34845 every 8 connection(s)
I20260619 12:34:03.589829 20464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-2/data/info.pb
I20260619 12:34:03.591517 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 20464
I20260619 12:34:03.591619 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-2/wal/instance
I20260619 12:34:03.591928 392 external_mini_cluster.cc:1428] Setting key 5776ab7d0c30d3fc1bd575dbd3c5bb75
I20260619 12:34:03.599493 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:03.591689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41435 (local address 127.0.98.62:46307)
0619 12:34:03.591854 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:03.591860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:03.593053 (+ 1193us) server_negotiation.cc:407] Connection header received
0619 12:34:03.594792 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:03.594795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:03.594850 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:03.594958 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:03.595833 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.596406 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.597633 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.597863 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.597973 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:03.598997 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:03.599000 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:03.599003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:03.599024 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:03.599042 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:03.599050 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:03.599129 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:03.599292 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 12:34:03.599353 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 12:34:03.600304 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:03.592142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46307 (local address 127.0.98.3:41435)
0619 12:34:03.592897 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:03.592951 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:03.594571 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:03.595127 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:03.595133 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:03.595218 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:03.595689 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:03.595701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.596533 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.596536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:03.597511 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:03.597520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.598007 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.598010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:03.598173 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:03.598727 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 12:34:03.598741 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:03.598781 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 12:34:03.598841 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:03.599131 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:03.599141 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:03.599150 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 12:34:03.599281 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 12:34:03.599550 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":111,"threads_started":1}
I20260619 12:34:03.601464 20592 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46307
I20260619 12:34:03.601734 20592 heartbeater.cc:464] Registering TS with master...
I20260619 12:34:03.602334 20592 heartbeater.cc:511] Master 127.0.98.62:46307 requested a full tablet report, sending...
I20260619 12:34:03.603440 20142 ts_manager.cc:194] Registered new tserver with Master: 339aaf3bc0654d94b7e89b3ef4614c04 (127.0.98.3:34845)
I20260619 12:34:03.604030 20142 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:41435
I20260619 12:34:03.606132 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 12:34:03.611763 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:03.608196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47900 (local address 127.0.98.62:46307)
0619 12:34:03.608320 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:03.608324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:03.608476 (+ 152us) server_negotiation.cc:407] Connection header received
0619 12:34:03.608579 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:03.608580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:03.608619 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:03.608685 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:03.608925 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.609501 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.610344 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.610551 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.610633 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:03.611414 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:03.611416 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:03.611418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:03.611428 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:03.611443 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:03.611457 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:03.611510 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:03.611600 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 12:34:03.611650 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 12:34:03.614830 20142 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 12:34:03.617982 20142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 12:34:03.632969 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:03.627928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52086 (local address 127.0.98.1:44107)
0619 12:34:03.628319 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:03.628330 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:03.628352 (+ 22us) server_negotiation.cc:407] Connection header received
0619 12:34:03.628427 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:03.628433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:03.628611 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:03.628801 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:34:03.629026 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.630146 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.631600 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.632128 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.632256 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:03.632452 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:03.632662 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 12:34:03.632769 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":170,"threads_started":1}
I20260619 12:34:03.632964 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:03.627331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44107 (local address 127.0.0.1:52086)
0619 12:34:03.627873 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:03.627885 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:03.627956 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:03.628805 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:03.628810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:03.628834 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:34:03.628909 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:03.628918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.630264 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.630267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:03.631505 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:03.631515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.632256 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.632260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:03.632454 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:03.632510 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 12:34:03.632617 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 12:34:03.632771 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":120,"thread_start_us":90,"threads_started":1}
I20260619 12:34:03.632974 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:03.626613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36829 (local address 127.0.0.1:37952)
0619 12:34:03.627450 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:03.627498 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:03.627636 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:03.628593 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:03.628597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:03.628615 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:34:03.628718 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:03.628725 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.629851 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.629857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:03.631430 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:03.631442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.632245 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.632248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:03.632436 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:03.632498 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 12:34:03.632603 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 12:34:03.632808 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":364,"spinlock_wait_cycles":12928,"thread_start_us":101,"threads_started":1}
I20260619 12:34:03.633445 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:03.627546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37952 (local address 127.0.98.2:36829)
0619 12:34:03.628215 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:03.628228 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:03.628254 (+ 26us) server_negotiation.cc:407] Connection header received
0619 12:34:03.628329 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:03.628332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:03.628461 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:03.628612 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:34:03.628876 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.629732 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.631559 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.632104 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.632232 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:03.632445 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:03.633177 (+ 732us) server_negotiation.cc:299] Negotiation successful
0619 12:34:03.633296 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":135,"threads_started":1}
I20260619 12:34:03.634663 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:03.626578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34845 (local address 127.0.0.1:57554)
0619 12:34:03.627446 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:03.627498 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:03.627636 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:03.628467 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:03.628473 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:03.628495 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:34:03.628623 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:03.628632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.630192 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.630199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:03.631438 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:03.631448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.634258 (+ 2810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.634262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:03.634350 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:03.634373 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:34:03.634454 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 12:34:03.634523 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":35,"spinlock_wait_cycles":1280,"thread_start_us":88,"threads_started":1}
I20260619 12:34:03.634773 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:03.626736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.0.98.3:34845)
0619 12:34:03.627064 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:03.627076 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:03.627779 (+ 703us) server_negotiation.cc:407] Connection header received
0619 12:34:03.627866 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:03.627872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:03.628108 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:03.628293 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:34:03.628789 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.630041 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.632505 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.633920 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.634023 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:03.634191 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:03.634511 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 12:34:03.634614 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":98,"threads_started":1}
I20260619 12:34:03.636310 20395 tablet_service.cc:1467] Processing CreateTablet for tablet aa57ab31e8ac4f1fba17eb2b9d838f7f (DEFAULT_TABLE table=test-table [id=59067c585299473fa3cd73075aafebd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:34:03.636336 20264 tablet_service.cc:1467] Processing CreateTablet for tablet aa57ab31e8ac4f1fba17eb2b9d838f7f (DEFAULT_TABLE table=test-table [id=59067c585299473fa3cd73075aafebd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:34:03.637389 20395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa57ab31e8ac4f1fba17eb2b9d838f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:03.637392 20264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa57ab31e8ac4f1fba17eb2b9d838f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:03.637635 20526 tablet_service.cc:1467] Processing CreateTablet for tablet aa57ab31e8ac4f1fba17eb2b9d838f7f (DEFAULT_TABLE table=test-table [id=59067c585299473fa3cd73075aafebd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 12:34:03.638532 20526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa57ab31e8ac4f1fba17eb2b9d838f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:03.643721 20609 tablet_bootstrap.cc:492] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe: Bootstrap starting.
I20260619 12:34:03.645191 20610 tablet_bootstrap.cc:492] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04: Bootstrap starting.
I20260619 12:34:03.646099 20609 tablet_bootstrap.cc:654] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:03.646164 20611 tablet_bootstrap.cc:492] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 4163003a226d42b6b7b49b210e771e2b: Bootstrap starting.
I20260619 12:34:03.647085 20609 log.cc:824] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:03.647639 20610 tablet_bootstrap.cc:654] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:03.648067 20611 tablet_bootstrap.cc:654] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 4163003a226d42b6b7b49b210e771e2b: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:03.648461 20610 log.cc:824] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:03.648782 20611 log.cc:824] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 4163003a226d42b6b7b49b210e771e2b: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:03.649868 20609 tablet_bootstrap.cc:492] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe: No bootstrap required, opened a new log
I20260619 12:34:03.650087 20609 ts_tablet_manager.cc:1397] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 12:34:03.650475 20610 tablet_bootstrap.cc:492] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04: No bootstrap required, opened a new log
I20260619 12:34:03.650664 20611 tablet_bootstrap.cc:492] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 4163003a226d42b6b7b49b210e771e2b: No bootstrap required, opened a new log
I20260619 12:34:03.650655 20610 ts_tablet_manager.cc:1397] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 12:34:03.650821 20611 ts_tablet_manager.cc:1397] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 4163003a226d42b6b7b49b210e771e2b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 12:34:03.653545 20609 raft_consensus.cc:348] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339aaf3bc0654d94b7e89b3ef4614c04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34845 } } peers { permanent_uuid: "cbbfa688bdc1413a93e15114c2a334fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36829 } } peers { permanent_uuid: "4163003a226d42b6b7b49b210e771e2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44107 } }
I20260619 12:34:03.653792 20609 raft_consensus.cc:374] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:03.653847 20609 raft_consensus.cc:729] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbbfa688bdc1413a93e15114c2a334fe, State: Initialized, Role: FOLLOWER
I20260619 12:34:03.654289 20609 consensus_queue.cc:260] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339aaf3bc0654d94b7e89b3ef4614c04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34845 } } peers { permanent_uuid: "cbbfa688bdc1413a93e15114c2a334fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36829 } } peers { permanent_uuid: "4163003a226d42b6b7b49b210e771e2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44107 } }
I20260619 12:34:03.654958 20461 heartbeater.cc:503] Master 127.0.98.62:46307 was elected leader, sending a full tablet report...
I20260619 12:34:03.654924 20611 raft_consensus.cc:348] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 4163003a226d42b6b7b49b210e771e2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339aaf3bc0654d94b7e89b3ef4614c04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34845 } } peers { permanent_uuid: "cbbfa688bdc1413a93e15114c2a334fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36829 } } peers { permanent_uuid: "4163003a226d42b6b7b49b210e771e2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44107 } }
I20260619 12:34:03.655200 20611 raft_consensus.cc:374] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 4163003a226d42b6b7b49b210e771e2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:03.655256 20611 raft_consensus.cc:729] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 4163003a226d42b6b7b49b210e771e2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4163003a226d42b6b7b49b210e771e2b, State: Initialized, Role: FOLLOWER
I20260619 12:34:03.655335 20609 ts_tablet_manager.cc:1428] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 12:34:03.655666 20611 consensus_queue.cc:260] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 4163003a226d42b6b7b49b210e771e2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339aaf3bc0654d94b7e89b3ef4614c04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34845 } } peers { permanent_uuid: "cbbfa688bdc1413a93e15114c2a334fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36829 } } peers { permanent_uuid: "4163003a226d42b6b7b49b210e771e2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44107 } }
I20260619 12:34:03.655694 20610 raft_consensus.cc:348] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339aaf3bc0654d94b7e89b3ef4614c04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34845 } } peers { permanent_uuid: "cbbfa688bdc1413a93e15114c2a334fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36829 } } peers { permanent_uuid: "4163003a226d42b6b7b49b210e771e2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44107 } }
I20260619 12:34:03.655932 20610 raft_consensus.cc:374] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:03.655980 20610 raft_consensus.cc:729] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 339aaf3bc0654d94b7e89b3ef4614c04, State: Initialized, Role: FOLLOWER
I20260619 12:34:03.656404 20610 consensus_queue.cc:260] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339aaf3bc0654d94b7e89b3ef4614c04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34845 } } peers { permanent_uuid: "cbbfa688bdc1413a93e15114c2a334fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36829 } } peers { permanent_uuid: "4163003a226d42b6b7b49b210e771e2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44107 } }
I20260619 12:34:03.656536 20611 ts_tablet_manager.cc:1428] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 4163003a226d42b6b7b49b210e771e2b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 12:34:03.656845 20330 heartbeater.cc:503] Master 127.0.98.62:46307 was elected leader, sending a full tablet report...
I20260619 12:34:03.657310 20610 ts_tablet_manager.cc:1428] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260619 12:34:03.657646 20592 heartbeater.cc:503] Master 127.0.98.62:46307 was elected leader, sending a full tablet report...
I20260619 12:34:03.659678 20615 raft_consensus.cc:482] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 12:34:03.659835 20615 raft_consensus.cc:504] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339aaf3bc0654d94b7e89b3ef4614c04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34845 } } peers { permanent_uuid: "cbbfa688bdc1413a93e15114c2a334fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36829 } } peers { permanent_uuid: "4163003a226d42b6b7b49b210e771e2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44107 } }
I20260619 12:34:03.661067 20615 leader_election.cc:288] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 339aaf3bc0654d94b7e89b3ef4614c04 (127.0.98.3:34845), 4163003a226d42b6b7b49b210e771e2b (127.0.98.1:44107)
I20260619 12:34:03.664625 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:03.661308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44107 (local address 127.0.98.2:50433)
0619 12:34:03.661414 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:03.661430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:03.661513 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:03.661935 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:03.661938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:03.661952 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:34:03.662044 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:03.662049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.662786 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.662790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:03.663558 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:03.663567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.664197 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.664201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:03.664311 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:03.664334 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 12:34:03.664422 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 12:34:03.664496 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 12:34:03.664654 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:03.661401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50433 (local address 127.0.98.1:44107)
0619 12:34:03.661506 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:03.661511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:03.661524 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:34:03.661610 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:03.661615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:03.661682 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:03.661748 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:34:03.662158 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.662658 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.663686 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.664072 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.664197 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:03.664256 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:03.664438 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 12:34:03.664554 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 12:34:03.664886 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:03.661724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52209 (local address 127.0.98.3:34845)
0619 12:34:03.661863 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:03.661867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:03.661879 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:34:03.661992 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:03.661995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:03.662032 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:03.662116 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 12:34:03.662393 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.662877 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.663929 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.664328 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.664447 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:03.664498 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:03.664696 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 12:34:03.664773 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 12:34:03.664951 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:03.661337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34845 (local address 127.0.98.2:52209)
0619 12:34:03.661788 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:03.661801 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:03.661887 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:03.662149 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:03.662151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:03.662167 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 12:34:03.662290 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:03.662295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.662985 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.662991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:03.663809 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:03.663818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.664458 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.664461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:03.664556 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:03.664577 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 12:34:03.664670 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 12:34:03.664815 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":3,"thread_start_us":120,"threads_started":1}
I20260619 12:34:03.665361 20284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa57ab31e8ac4f1fba17eb2b9d838f7f" candidate_uuid: "cbbfa688bdc1413a93e15114c2a334fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4163003a226d42b6b7b49b210e771e2b" is_pre_election: true
I20260619 12:34:03.665489 20546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa57ab31e8ac4f1fba17eb2b9d838f7f" candidate_uuid: "cbbfa688bdc1413a93e15114c2a334fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "339aaf3bc0654d94b7e89b3ef4614c04" is_pre_election: true
I20260619 12:34:03.665681 20284 raft_consensus.cc:2454] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 4163003a226d42b6b7b49b210e771e2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbbfa688bdc1413a93e15114c2a334fe in term 0.
I20260619 12:34:03.665699 20546 raft_consensus.cc:2454] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbbfa688bdc1413a93e15114c2a334fe in term 0.
I20260619 12:34:03.666126 20349 leader_election.cc:302] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4163003a226d42b6b7b49b210e771e2b, cbbfa688bdc1413a93e15114c2a334fe; no voters:
I20260619 12:34:03.666404 20615 raft_consensus.cc:2781] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 12:34:03.666486 20615 raft_consensus.cc:482] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 12:34:03.666528 20615 raft_consensus.cc:3037] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:03.667567 20615 raft_consensus.cc:504] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339aaf3bc0654d94b7e89b3ef4614c04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34845 } } peers { permanent_uuid: "cbbfa688bdc1413a93e15114c2a334fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36829 } } peers { permanent_uuid: "4163003a226d42b6b7b49b210e771e2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44107 } }
I20260619 12:34:03.667990 20615 leader_election.cc:288] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [CANDIDATE]: Term 1 election: Requested vote from peers 339aaf3bc0654d94b7e89b3ef4614c04 (127.0.98.3:34845), 4163003a226d42b6b7b49b210e771e2b (127.0.98.1:44107)
I20260619 12:34:03.668349 20284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa57ab31e8ac4f1fba17eb2b9d838f7f" candidate_uuid: "cbbfa688bdc1413a93e15114c2a334fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4163003a226d42b6b7b49b210e771e2b"
I20260619 12:34:03.668349 20546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa57ab31e8ac4f1fba17eb2b9d838f7f" candidate_uuid: "cbbfa688bdc1413a93e15114c2a334fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "339aaf3bc0654d94b7e89b3ef4614c04"
I20260619 12:34:03.668458 20546 raft_consensus.cc:3037] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:03.668458 20284 raft_consensus.cc:3037] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 4163003a226d42b6b7b49b210e771e2b [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:03.669795 20284 raft_consensus.cc:2454] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 4163003a226d42b6b7b49b210e771e2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbbfa688bdc1413a93e15114c2a334fe in term 1.
I20260619 12:34:03.669795 20546 raft_consensus.cc:2454] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbbfa688bdc1413a93e15114c2a334fe in term 1.
I20260619 12:34:03.670109 20350 leader_election.cc:302] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 339aaf3bc0654d94b7e89b3ef4614c04, cbbfa688bdc1413a93e15114c2a334fe; no voters:
I20260619 12:34:03.670329 20615 raft_consensus.cc:2781] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:03.670573 20615 raft_consensus.cc:686] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [term 1 LEADER]: Becoming Leader. State: Replica: cbbfa688bdc1413a93e15114c2a334fe, State: Running, Role: LEADER
I20260619 12:34:03.670897 20615 consensus_queue.cc:237] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339aaf3bc0654d94b7e89b3ef4614c04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34845 } } peers { permanent_uuid: "cbbfa688bdc1413a93e15114c2a334fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36829 } } peers { permanent_uuid: "4163003a226d42b6b7b49b210e771e2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44107 } }
I20260619 12:34:03.674157 20142 catalog_manager.cc:5495] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe reported cstate change: term changed from 0 to 1, leader changed from <none> to cbbfa688bdc1413a93e15114c2a334fe (127.0.98.2). New cstate: current_term: 1 leader_uuid: "cbbfa688bdc1413a93e15114c2a334fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339aaf3bc0654d94b7e89b3ef4614c04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbbfa688bdc1413a93e15114c2a334fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4163003a226d42b6b7b49b210e771e2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44107 } health_report { overall_health: UNKNOWN } } }
I20260619 12:34:03.686499 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:03.682765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37960 (local address 127.0.98.2:36829)
0619 12:34:03.682892 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:03.682896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:03.682909 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:34:03.683062 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:03.683064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:03.683116 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:03.683204 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:34:03.683408 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.683892 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.684765 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:03.684982 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:03.685251 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:03.685554 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:34:03.685982 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:34:03.686087 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:03.686210 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 12:34:03.686259 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 12:34:03.689853 20331 tablet.cc:2369] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 4163003a226d42b6b7b49b210e771e2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:34:03.692502 20284 raft_consensus.cc:1261] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 4163003a226d42b6b7b49b210e771e2b [term 1 FOLLOWER]: Refusing update from remote peer cbbfa688bdc1413a93e15114c2a334fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:34:03.692502 20546 raft_consensus.cc:1261] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04 [term 1 FOLLOWER]: Refusing update from remote peer cbbfa688bdc1413a93e15114c2a334fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:34:03.693293 20619 consensus_queue.cc:1035] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4163003a226d42b6b7b49b210e771e2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:34:03.693475 20615 consensus_queue.cc:1035] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "339aaf3bc0654d94b7e89b3ef4614c04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:34:03.701994 20623 mvcc.cc:205] Tried to move back new op lower bound from 7298549529357561856 to 7298549529278615552. Current Snapshot: MvccSnapshot[applied={T|T < 7298549529357561856}]
I20260619 12:34:03.702193 20624 mvcc.cc:205] Tried to move back new op lower bound from 7298549529357561856 to 7298549529278615552. Current Snapshot: MvccSnapshot[applied={T|T < 7298549529357561856}]
I20260619 12:34:03.712070 20141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:34:03.712255 20141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 12:34:03.714823 20141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a173c93cbf1049ee804412dc3e543a37: Sending DeleteTablet for 3 replicas of tablet aa57ab31e8ac4f1fba17eb2b9d838f7f
I20260619 12:34:03.715678 20526 tablet_service.cc:1514] Processing DeleteTablet for tablet aa57ab31e8ac4f1fba17eb2b9d838f7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:34:03 UTC) from {username='slave'} at 127.0.0.1:57554
I20260619 12:34:03.715727 20395 tablet_service.cc:1514] Processing DeleteTablet for tablet aa57ab31e8ac4f1fba17eb2b9d838f7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:34:03 UTC) from {username='slave'} at 127.0.0.1:37952
I20260619 12:34:03.715749 20264 tablet_service.cc:1514] Processing DeleteTablet for tablet aa57ab31e8ac4f1fba17eb2b9d838f7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 12:34:03 UTC) from {username='slave'} at 127.0.0.1:52086
I20260619 12:34:03.716207 20633 tablet_replica.cc:330] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe: stopping tablet replica
I20260619 12:34:03.716254 20634 tablet_replica.cc:330] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 4163003a226d42b6b7b49b210e771e2b: stopping tablet replica
I20260619 12:34:03.716531 20633 raft_consensus.cc:2229] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [term 1 LEADER]: Raft consensus shutting down.
I20260619 12:34:03.716562 20634 raft_consensus.cc:2229] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 4163003a226d42b6b7b49b210e771e2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:34:03.716886 20634 raft_consensus.cc:2258] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 4163003a226d42b6b7b49b210e771e2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:34:03.716951 20633 raft_consensus.cc:2258] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:34:03.717417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 20202
I20260619 12:34:03.718144 20632 tablet_replica.cc:330] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04: stopping tablet replica
I20260619 12:34:03.718446 20632 raft_consensus.cc:2229] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 12:34:03.718744 20632 raft_consensus.cc:2258] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 12:34:03.720768 20633 ts_tablet_manager.cc:1905] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:34:03.724418 20633 ts_tablet_manager.cc:1918] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:34:03.724536 20633 log.cc:1196] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-1/wal/wals/aa57ab31e8ac4f1fba17eb2b9d838f7f
I20260619 12:34:03.724879 20633 ts_tablet_manager.cc:1939] T aa57ab31e8ac4f1fba17eb2b9d838f7f P cbbfa688bdc1413a93e15114c2a334fe: Deleting consensus metadata
I20260619 12:34:03.726282 20127 catalog_manager.cc:4841] TS cbbfa688bdc1413a93e15114c2a334fe (127.0.98.2:36829): tablet aa57ab31e8ac4f1fba17eb2b9d838f7f (table test-table [id=59067c585299473fa3cd73075aafebd5]) successfully deleted
W20260619 12:34:03.726824 20127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44107 (error 108)
I20260619 12:34:03.726955 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 20333
I20260619 12:34:03.728322 20632 ts_tablet_manager.cc:1905] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 12:34:03.728744 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:03.728429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44107 (local address 127.0.0.1:52102)
0619 12:34:03.728597 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:03.728660 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260619 12:34:03.728960 20127 catalog_manager.cc:4584] TS 4163003a226d42b6b7b49b210e771e2b (127.0.98.1:44107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa57ab31e8ac4f1fba17eb2b9d838f7f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44107: connect: Connection refused (error 111)
I20260619 12:34:03.730580 20632 ts_tablet_manager.cc:1918] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 12:34:03.730700 20632 log.cc:1196] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04: Deleting WAL directory at /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781872188319340-392-0/minicluster-data/ts-2/wal/wals/aa57ab31e8ac4f1fba17eb2b9d838f7f
I20260619 12:34:03.730967 20632 ts_tablet_manager.cc:1939] T aa57ab31e8ac4f1fba17eb2b9d838f7f P 339aaf3bc0654d94b7e89b3ef4614c04: Deleting consensus metadata
I20260619 12:34:03.732040 20128 catalog_manager.cc:4841] TS 339aaf3bc0654d94b7e89b3ef4614c04 (127.0.98.3:34845): tablet aa57ab31e8ac4f1fba17eb2b9d838f7f (table test-table [id=59067c585299473fa3cd73075aafebd5]) successfully deleted
I20260619 12:34:03.732762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 20464
I20260619 12:34:03.738196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 20111
2026-06-19T12:34:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1002 ms)
[----------] 6 tests from EncryptionPolicyTest (10489 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:03 dist-test-slave-fg3h krb5kdc[20638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:03 dist-test-slave-fg3h krb5kdc[20638](info): set up 2 sockets
Jun 19 12:34:03 dist-test-slave-fg3h krb5kdc[20638](info): commencing operation
krb5kdc: starting...
W20260619 12:34:04.781255 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:04 dist-test-slave-fg3h krb5kdc[20638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:34:04.902158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44117
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:05.013451 20651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:05.013783 20651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:05.013871 20651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:05.018069 20651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:05.018157 20651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:05.018194 20651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:05.018260 20651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:05.018316 20651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:05.023968 20651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44117
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:05.025189 20651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:05.026284 20651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:05.035166 20657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:05.035176 20659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:05.035285 20651 server_base.cc:1034] running on GCE node
W20260619 12:34:05.035451 20656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:05.035965 20651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:05.037177 20651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:05.038378 20651 hybrid_clock.cc:648] HybridClock initialized: now 1781872445038354 us; error 47 us; skew 500 ppm
Jun 19 12:34:05 dist-test-slave-fg3h krb5kdc[20638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:05.043948 20651 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:34:05.044874 20651 webserver.cc:460] Webserver started at http://127.0.0.1:37329/ using document root <none> and password file <none>
I20260619 12:34:05.045547 20651 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:05.045647 20651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:05.045931 20651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:05.048004 20651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "d78bc3129d3a4ec19ae5d676b0d8813f"
format_stamp: "Formatted at 2026-06-19 12:34:05 on dist-test-slave-fg3h"
server_key: "ec0af9b0f3888ef5fdf0205dfd14b975"
server_key_iv: "000edd7d609a2421dc234cbb23dcb981"
server_key_version: "encryptionkey@0"
I20260619 12:34:05.048601 20651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "d78bc3129d3a4ec19ae5d676b0d8813f"
format_stamp: "Formatted at 2026-06-19 12:34:05 on dist-test-slave-fg3h"
server_key: "ec0af9b0f3888ef5fdf0205dfd14b975"
server_key_iv: "000edd7d609a2421dc234cbb23dcb981"
server_key_version: "encryptionkey@0"
I20260619 12:34:05.052701 20651 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 12:34:05.056867 20666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:05.059165 20651 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:34:05.059324 20651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "d78bc3129d3a4ec19ae5d676b0d8813f"
format_stamp: "Formatted at 2026-06-19 12:34:05 on dist-test-slave-fg3h"
server_key: "ec0af9b0f3888ef5fdf0205dfd14b975"
server_key_iv: "000edd7d609a2421dc234cbb23dcb981"
server_key_version: "encryptionkey@0"
I20260619 12:34:05.059470 20651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:05.109023 20651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:05.112524 20651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:05.112692 20651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:05.119699 20651 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44117
I20260619 12:34:05.119767 20718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44117 every 8 connection(s)
I20260619 12:34:05.120690 20651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:05.123576 20719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:05.129128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 20651
I20260619 12:34:05.129276 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:05.129285 20719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f: Bootstrap starting.
I20260619 12:34:05.129612 392 external_mini_cluster.cc:1428] Setting key c620d39ad9a2a4dfd7da0a77d73e935f
I20260619 12:34:05.131873 20719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:05.132601 20719 log.cc:824] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:05.134495 20719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f: No bootstrap required, opened a new log
I20260619 12:34:05.137960 20719 raft_consensus.cc:348] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78bc3129d3a4ec19ae5d676b0d8813f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44117 } }
I20260619 12:34:05.138265 20719 raft_consensus.cc:374] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:05.138330 20719 raft_consensus.cc:729] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d78bc3129d3a4ec19ae5d676b0d8813f, State: Initialized, Role: FOLLOWER
Jun 19 12:34:05 dist-test-slave-fg3h krb5kdc[20638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 12:34:05.138820 20719 consensus_queue.cc:260] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78bc3129d3a4ec19ae5d676b0d8813f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44117 } }
I20260619 12:34:05.138998 20719 raft_consensus.cc:388] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:05.139084 20719 raft_consensus.cc:482] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:05.139217 20719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:05.140477 20719 raft_consensus.cc:504] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78bc3129d3a4ec19ae5d676b0d8813f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44117 } }
I20260619 12:34:05.140846 20719 leader_election.cc:302] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d78bc3129d3a4ec19ae5d676b0d8813f; no voters:
I20260619 12:34:05.141243 20719 leader_election.cc:288] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:05.141340 20724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:05.141578 20724 raft_consensus.cc:686] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f [term 1 LEADER]: Becoming Leader. State: Replica: d78bc3129d3a4ec19ae5d676b0d8813f, State: Running, Role: LEADER
I20260619 12:34:05.141935 20724 consensus_queue.cc:237] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78bc3129d3a4ec19ae5d676b0d8813f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44117 } }
I20260619 12:34:05.142465 20719 sys_catalog.cc:564] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:05.144047 20725 sys_catalog.cc:455] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d78bc3129d3a4ec19ae5d676b0d8813f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78bc3129d3a4ec19ae5d676b0d8813f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44117 } } }
I20260619 12:34:05.144184 20725 sys_catalog.cc:458] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:05.144588 20726 sys_catalog.cc:455] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d78bc3129d3a4ec19ae5d676b0d8813f. Latest consensus state: current_term: 1 leader_uuid: "d78bc3129d3a4ec19ae5d676b0d8813f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78bc3129d3a4ec19ae5d676b0d8813f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44117 } } }
I20260619 12:34:05.144994 20726 sys_catalog.cc:458] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:05.145430 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:05.131034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46897 (local address 127.0.0.1:44117)
0619 12:34:05.131461 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:05.131476 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:05.131511 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:34:05.132220 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:05.132239 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:05.132589 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:05.132962 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:05.133793 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:05.134847 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:05.135709 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:05.136193 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:05.136372 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:05.139283 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:05.139291 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:05.139311 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:05.139343 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:05.141376 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:05.142103 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:05.142109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:05.142114 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:05.142313 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:05.142746 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:05.142750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:05.142752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:05.143296 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:05.143534 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:05.144177 (+ 643us) server_negotiation.cc:299] Negotiation successful
0619 12:34:05.144409 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":124,"threads_started":1}
W20260619 12:34:05.147650 20739 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:34:05.147720 20739 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:34:05.147782 20736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:05.148694 20736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:05.148931 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:34:05.154479 20736 catalog_manager.cc:1347] Generated new cluster ID: 8803531e94be4f86bb0074e314cd7d9d
I20260619 12:34:05.154546 20736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:05.159965 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:05.151273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39375 (local address 127.0.0.1:44117)
0619 12:34:05.151479 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:05.151483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:05.151731 (+ 248us) server_negotiation.cc:407] Connection header received
0619 12:34:05.151833 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:05.151837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:05.151883 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:05.151969 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:05.152845 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:05.153687 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:05.154783 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:05.155033 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:05.155161 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:05.156568 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:05.156572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:05.156575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:05.156599 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:05.158136 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:05.158618 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:05.158621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:05.158634 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:05.158708 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:05.159085 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:05.159088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:05.159089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:05.159262 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:05.159445 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:05.159688 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 12:34:05.159783 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260619 12:34:05.204972 20736 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:05.206359 20736 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:05.214464 20736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d78bc3129d3a4ec19ae5d676b0d8813f: Generated new TSK 0
I20260619 12:34:05.215301 20736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:34:05.230744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 20651
W20260619 12:34:05.236291 20636 connection.cc:535] client connection to 127.0.0.1:44117 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44117): Connection reset by peer (error 104)
2026-06-19T12:34:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1491 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:05 dist-test-slave-fg3h krb5kdc[20750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:05 dist-test-slave-fg3h krb5kdc[20750](info): set up 2 sockets
Jun 19 12:34:05 dist-test-slave-fg3h krb5kdc[20750](info): commencing operation
krb5kdc: starting...
W20260619 12:34:06.277297 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:06 dist-test-slave-fg3h krb5kdc[20750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:34:06.400349 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:06.509598 20763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:06.509933 20763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:06.510012 20763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:06.514175 20763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:06.514284 20763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:06.514323 20763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:06.514380 20763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:06.514418 20763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:06.519929 20763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33849
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:06.521077 20763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:06.522243 20763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:06.531304 20769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:06.531319 20768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:06.531325 20771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:06.531431 20763 server_base.cc:1034] running on GCE node
I20260619 12:34:06.532223 20763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:06.533511 20763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:06.534732 20763 hybrid_clock.cc:648] HybridClock initialized: now 1781872446534696 us; error 50 us; skew 500 ppm
Jun 19 12:34:06 dist-test-slave-fg3h krb5kdc[20750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:06.537686 20763 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:34:06.538658 20763 webserver.cc:460] Webserver started at http://127.0.0.1:37243/ using document root <none> and password file <none>
I20260619 12:34:06.539279 20763 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:06.539381 20763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:06.539616 20763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:06.541812 20763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "96d4cc92830743218a18216177e94182"
format_stamp: "Formatted at 2026-06-19 12:34:06 on dist-test-slave-fg3h"
server_key: "74774372b8627a342c0b70514af10575"
server_key_iv: "1d12275a48168b2f0be6537ab5f301d6"
server_key_version: "encryptionkey@0"
I20260619 12:34:06.542415 20763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "96d4cc92830743218a18216177e94182"
format_stamp: "Formatted at 2026-06-19 12:34:06 on dist-test-slave-fg3h"
server_key: "74774372b8627a342c0b70514af10575"
server_key_iv: "1d12275a48168b2f0be6537ab5f301d6"
server_key_version: "encryptionkey@0"
I20260619 12:34:06.546320 20763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 12:34:06.548813 20778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:06.550172 20763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:34:06.550359 20763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "96d4cc92830743218a18216177e94182"
format_stamp: "Formatted at 2026-06-19 12:34:06 on dist-test-slave-fg3h"
server_key: "74774372b8627a342c0b70514af10575"
server_key_iv: "1d12275a48168b2f0be6537ab5f301d6"
server_key_version: "encryptionkey@0"
I20260619 12:34:06.550479 20763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:06.573359 20763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:06.576548 20763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:06.576767 20763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:06.583966 20763 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33849
I20260619 12:34:06.584024 20830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33849 every 8 connection(s)
I20260619 12:34:06.584939 20763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:06.586619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 20763
I20260619 12:34:06.586766 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:06.587082 392 external_mini_cluster.cc:1428] Setting key 5e5d69589248501e06215a7b60db2f5f
I20260619 12:34:06.588320 20831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:06.593966 20831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182: Bootstrap starting.
I20260619 12:34:06.596150 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:06.588497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55069 (local address 127.0.0.1:33849)
0619 12:34:06.589109 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:06.589210 (+ 101us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:06.589255 (+ 45us) server_negotiation.cc:407] Connection header received
0619 12:34:06.589897 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:06.589913 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:06.590255 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:06.590586 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:06.590978 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:06.591954 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:06.592841 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:06.593406 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:06.593576 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:06.594549 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:06.594553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:06.594566 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:06.594937 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:06.594989 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:06.595012 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:06.595157 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:06.595324 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 12:34:06.595455 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":159,"threads_started":1}
I20260619 12:34:06.596334 20831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:06.597025 20831 log.cc:824] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:06.599188 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:34:06.599272 20831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182: No bootstrap required, opened a new log
I20260619 12:34:06.603466 20831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d4cc92830743218a18216177e94182" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33849 } }
I20260619 12:34:06.603766 20831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:06.603833 20831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96d4cc92830743218a18216177e94182, State: Initialized, Role: FOLLOWER
I20260619 12:34:06.604295 20831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d4cc92830743218a18216177e94182" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33849 } }
I20260619 12:34:06.604468 20831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:06.604550 20831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:06.604665 20831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:06.605788 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:06.601227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42289 (local address 127.0.0.1:33849)
0619 12:34:06.601344 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:06.601348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:06.601629 (+ 281us) server_negotiation.cc:407] Connection header received
0619 12:34:06.601723 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:06.601725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:06.601762 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:06.601831 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:06.602126 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:06.602617 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:06.603777 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:06.604132 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:06.604226 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:06.605251 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:06.605254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:06.605256 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:06.605267 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:06.605292 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:06.605302 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:06.605381 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:06.605529 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 12:34:06.605568 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 12:34:06.605957 20831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d4cc92830743218a18216177e94182" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33849 } }
I20260619 12:34:06.606328 20831 leader_election.cc:302] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96d4cc92830743218a18216177e94182; no voters:
I20260619 12:34:06.606662 20831 leader_election.cc:288] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:06.606758 20841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:06.606992 20841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182 [term 1 LEADER]: Becoming Leader. State: Replica: 96d4cc92830743218a18216177e94182, State: Running, Role: LEADER
I20260619 12:34:06.607295 20841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d4cc92830743218a18216177e94182" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33849 } }
I20260619 12:34:06.607867 20831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:06.608835 20842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96d4cc92830743218a18216177e94182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d4cc92830743218a18216177e94182" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33849 } } }
I20260619 12:34:06.608990 20842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:06.609196 20843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96d4cc92830743218a18216177e94182. Latest consensus state: current_term: 1 leader_uuid: "96d4cc92830743218a18216177e94182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d4cc92830743218a18216177e94182" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33849 } } }
I20260619 12:34:06.609292 20843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:06.610440 20850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:06.613322 20850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:06.618937 20850 catalog_manager.cc:1347] Generated new cluster ID: e59747e58e1442d89553798d2ac7529d
I20260619 12:34:06.619022 20850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:06.653138 20850 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:06.654258 20850 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:06.668718 20850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96d4cc92830743218a18216177e94182: Generated new TSK 0
I20260619 12:34:06.669497 20850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:34:06.675254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 20763
W20260619 12:34:06.680757 20748 connection.cc:535] client connection to 127.0.0.1:33849 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33849): Connection reset by peer (error 104)
2026-06-19T12:34:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:06 dist-test-slave-fg3h krb5kdc[20862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:06 dist-test-slave-fg3h krb5kdc[20862](info): set up 2 sockets
Jun 19 12:34:06 dist-test-slave-fg3h krb5kdc[20862](info): commencing operation
krb5kdc: starting...
W20260619 12:34:07.713346 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:07 dist-test-slave-fg3h krb5kdc[20862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:34:07.845307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39673
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:07.955231 20875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:07.955578 20875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:07.955680 20875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:07.960233 20875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:07.960300 20875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:07.960320 20875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:07.960336 20875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:07.960388 20875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:07.965767 20875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39673
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:07.966786 20875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:07.967803 20875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:07.976697 20880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:07.976709 20881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:07.976719 20883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:07.976912 20875 server_base.cc:1034] running on GCE node
I20260619 12:34:07.977674 20875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:07.978904 20875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:07.980139 20875 hybrid_clock.cc:648] HybridClock initialized: now 1781872447980064 us; error 51 us; skew 500 ppm
Jun 19 12:34:07 dist-test-slave-fg3h krb5kdc[20862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:07.983170 20875 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:34:07.984153 20875 webserver.cc:460] Webserver started at http://127.0.0.1:41159/ using document root <none> and password file <none>
I20260619 12:34:07.984745 20875 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:07.984808 20875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:07.985029 20875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:07.987164 20875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "0c5671eecd794f62a064873cfbab4737"
format_stamp: "Formatted at 2026-06-19 12:34:07 on dist-test-slave-fg3h"
server_key: "cad2e071093725e3b507cddc65d18358"
server_key_iv: "a90ada47fbc2ac5b8ac4fe6e991d2526"
server_key_version: "encryptionkey@0"
I20260619 12:34:07.987761 20875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c5671eecd794f62a064873cfbab4737"
format_stamp: "Formatted at 2026-06-19 12:34:07 on dist-test-slave-fg3h"
server_key: "cad2e071093725e3b507cddc65d18358"
server_key_iv: "a90ada47fbc2ac5b8ac4fe6e991d2526"
server_key_version: "encryptionkey@0"
I20260619 12:34:07.991586 20875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:34:07.994009 20890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:07.995257 20875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:34:07.995393 20875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "0c5671eecd794f62a064873cfbab4737"
format_stamp: "Formatted at 2026-06-19 12:34:07 on dist-test-slave-fg3h"
server_key: "cad2e071093725e3b507cddc65d18358"
server_key_iv: "a90ada47fbc2ac5b8ac4fe6e991d2526"
server_key_version: "encryptionkey@0"
I20260619 12:34:07.995505 20875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:08.000067 20875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:08.003423 20875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:08.003628 20875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:08.012298 20875 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39673
I20260619 12:34:08.012331 20942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39673 every 8 connection(s)
I20260619 12:34:08.013897 20875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:08.018002 20943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:08.021989 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 20875
I20260619 12:34:08.022143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:08.022450 392 external_mini_cluster.cc:1428] Setting key e0f8ca5b231d0fc99f2de7f64ffba972
I20260619 12:34:08.024890 20943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737: Bootstrap starting.
I20260619 12:34:08.027717 20943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:08.028196 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:08.024032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60377 (local address 127.0.0.1:39673)
0619 12:34:08.024503 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:08.024519 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:08.024555 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:34:08.025345 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:08.025364 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:08.025672 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:08.026068 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:08.026929 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:08.026933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:08.026945 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:08.026956 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:08.027014 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:08.027033 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:08.027148 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:08.027274 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 12:34:08.027438 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":152,"threads_started":1}
I20260619 12:34:08.028640 20943 log.cc:824] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:08.030819 20943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737: No bootstrap required, opened a new log
I20260619 12:34:08.031172 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:34:08.034845 20943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5671eecd794f62a064873cfbab4737" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39673 } }
I20260619 12:34:08.035163 20943 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:08.035205 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:08.033462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57871 (local address 127.0.0.1:39673)
0619 12:34:08.033600 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:08.033607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:08.033821 (+ 214us) server_negotiation.cc:407] Connection header received
0619 12:34:08.033939 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:08.033941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:08.033968 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:08.034033 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:08.034787 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:08.034790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:08.034792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:08.034801 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:08.034827 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:08.034837 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:08.034916 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:08.035020 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 12:34:08.035051 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 12:34:08.035233 20943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c5671eecd794f62a064873cfbab4737, State: Initialized, Role: FOLLOWER
I20260619 12:34:08.035696 20943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5671eecd794f62a064873cfbab4737" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39673 } }
I20260619 12:34:08.035859 20943 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:08.035930 20943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:08.036036 20943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:08.036966 20943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5671eecd794f62a064873cfbab4737" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39673 } }
I20260619 12:34:08.037333 20943 leader_election.cc:302] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c5671eecd794f62a064873cfbab4737; no voters:
I20260619 12:34:08.037626 20943 leader_election.cc:288] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:08.037775 20953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:08.038023 20953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737 [term 1 LEADER]: Becoming Leader. State: Replica: 0c5671eecd794f62a064873cfbab4737, State: Running, Role: LEADER
I20260619 12:34:08.038334 20953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5671eecd794f62a064873cfbab4737" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39673 } }
I20260619 12:34:08.038858 20943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:08.039992 20954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c5671eecd794f62a064873cfbab4737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5671eecd794f62a064873cfbab4737" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39673 } } }
I20260619 12:34:08.040131 20954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:08.040447 20955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c5671eecd794f62a064873cfbab4737. Latest consensus state: current_term: 1 leader_uuid: "0c5671eecd794f62a064873cfbab4737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5671eecd794f62a064873cfbab4737" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39673 } } }
I20260619 12:34:08.040584 20955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:08.041322 20962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:08.044448 20962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:08.050161 20962 catalog_manager.cc:1347] Generated new cluster ID: d123dc0a3a444001a9bd52cbd4d87db5
I20260619 12:34:08.050233 20962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:08.059638 20962 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:08.060007 20962 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:08.074146 20962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c5671eecd794f62a064873cfbab4737: Generated new TSK 0
I20260619 12:34:08.074923 20962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:34:08.083835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 20875
2026-06-19T12:34:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:08 dist-test-slave-fg3h krb5kdc[20974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:08 dist-test-slave-fg3h krb5kdc[20974](info): set up 2 sockets
Jun 19 12:34:08 dist-test-slave-fg3h krb5kdc[20974](info): commencing operation
krb5kdc: starting...
W20260619 12:34:09.125378 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:09 dist-test-slave-fg3h krb5kdc[20974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:34:09.252007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40733
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:09.361737 20987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:09.362160 20987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:09.362238 20987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:09.366545 20987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:09.366611 20987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:09.366631 20987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:09.366647 20987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:09.366662 20987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:09.372014 20987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:09.373008 20987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:09.374053 20987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:09.382452 20993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:09.382551 20992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:09.382608 20987 server_base.cc:1034] running on GCE node
W20260619 12:34:09.382551 20995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:09.383284 20987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:09.384465 20987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:09.385669 20987 hybrid_clock.cc:648] HybridClock initialized: now 1781872449385655 us; error 26 us; skew 500 ppm
Jun 19 12:34:09 dist-test-slave-fg3h krb5kdc[20974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:09.393244 20987 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:34:09.394191 20987 webserver.cc:460] Webserver started at http://127.0.0.1:36491/ using document root <none> and password file <none>
I20260619 12:34:09.394752 20987 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:09.394814 20987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:09.394985 20987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:09.397058 20987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "661ff71e85ef4ae081df1c8abb8bb9fa"
format_stamp: "Formatted at 2026-06-19 12:34:09 on dist-test-slave-fg3h"
server_key: "25b44f0392e1c532673e11de6fa23b1a"
server_key_iv: "73f2944c38a4aaf77c3e9a6ebe7ac941"
server_key_version: "encryptionkey@0"
I20260619 12:34:09.397699 20987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "661ff71e85ef4ae081df1c8abb8bb9fa"
format_stamp: "Formatted at 2026-06-19 12:34:09 on dist-test-slave-fg3h"
server_key: "25b44f0392e1c532673e11de6fa23b1a"
server_key_iv: "73f2944c38a4aaf77c3e9a6ebe7ac941"
server_key_version: "encryptionkey@0"
I20260619 12:34:09.401449 20987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:34:09.403852 21002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:09.405172 20987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:34:09.405336 20987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "661ff71e85ef4ae081df1c8abb8bb9fa"
format_stamp: "Formatted at 2026-06-19 12:34:09 on dist-test-slave-fg3h"
server_key: "25b44f0392e1c532673e11de6fa23b1a"
server_key_iv: "73f2944c38a4aaf77c3e9a6ebe7ac941"
server_key_version: "encryptionkey@0"
I20260619 12:34:09.405443 20987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:09.420123 20987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:09.423581 20987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:09.423794 20987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:09.431727 20987 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40733
I20260619 12:34:09.431754 21054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40733 every 8 connection(s)
I20260619 12:34:09.432678 20987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:09.435374 21055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:09.438530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 20987
I20260619 12:34:09.438673 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:09.438987 392 external_mini_cluster.cc:1428] Setting key 0f9e6529b8cbef184d143bf445881130
I20260619 12:34:09.441496 21055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa: Bootstrap starting.
I20260619 12:34:09.443624 21055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:09.444351 21055 log.cc:824] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:09.446251 21055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa: No bootstrap required, opened a new log
Jun 19 12:34:09 dist-test-slave-fg3h krb5kdc[20974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 12:34:09.450081 21055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "661ff71e85ef4ae081df1c8abb8bb9fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40733 } }
I20260619 12:34:09.450297 21055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:09.450359 21055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 661ff71e85ef4ae081df1c8abb8bb9fa, State: Initialized, Role: FOLLOWER
I20260619 12:34:09.450866 21055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "661ff71e85ef4ae081df1c8abb8bb9fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40733 } }
I20260619 12:34:09.451048 21055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:09.451122 21055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:09.451241 21055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:09.452328 21055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "661ff71e85ef4ae081df1c8abb8bb9fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40733 } }
I20260619 12:34:09.452688 21055 leader_election.cc:302] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 661ff71e85ef4ae081df1c8abb8bb9fa; no voters:
I20260619 12:34:09.453049 21055 leader_election.cc:288] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:09.453146 21060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:09.453361 21060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa [term 1 LEADER]: Becoming Leader. State: Replica: 661ff71e85ef4ae081df1c8abb8bb9fa, State: Running, Role: LEADER
I20260619 12:34:09.453438 21058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:09.440426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40947 (local address 127.0.0.1:40733)
0619 12:34:09.440864 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:09.440878 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:09.440913 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:34:09.441674 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:09.441695 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:09.442044 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:09.442381 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:09.443157 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:09.444208 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:09.445124 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:09.445599 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:09.445777 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:09.448348 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:09.448352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:09.448366 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:09.448393 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:09.450244 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:09.450772 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:09.450776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:09.450780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:09.450890 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:09.451247 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:09.451250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:09.451251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:09.451687 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:09.451842 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:09.452128 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 12:34:09.452430 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":137,"threads_started":1}
I20260619 12:34:09.453696 21060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "661ff71e85ef4ae081df1c8abb8bb9fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40733 } }
I20260619 12:34:09.454339 21055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:09.454859 21062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 661ff71e85ef4ae081df1c8abb8bb9fa. Latest consensus state: current_term: 1 leader_uuid: "661ff71e85ef4ae081df1c8abb8bb9fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "661ff71e85ef4ae081df1c8abb8bb9fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40733 } } }
I20260619 12:34:09.454985 21062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:09.455668 21061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "661ff71e85ef4ae081df1c8abb8bb9fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "661ff71e85ef4ae081df1c8abb8bb9fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40733 } } }
I20260619 12:34:09.455801 21061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:09.456566 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 12:34:09.459674 21075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:34:09.459775 21075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:34:09.459885 21065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:09.460754 21065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:09.467255 21065 catalog_manager.cc:1347] Generated new cluster ID: ad961e39c08f4043afe8d95830c36cb3
I20260619 12:34:09.467325 21065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:09.467686 21058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:09.458875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41613 (local address 127.0.0.1:40733)
0619 12:34:09.458984 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:09.458990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:09.459232 (+ 242us) server_negotiation.cc:407] Connection header received
0619 12:34:09.459407 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:09.459412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:09.459457 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:09.459507 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:09.460426 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:09.461051 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:09.461961 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:09.462217 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:09.462345 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:09.464302 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:09.464306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:09.464309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:09.464334 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:09.465918 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:09.466335 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:09.466340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:09.466342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:09.466423 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:09.466700 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:09.466703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:09.466705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:09.466915 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:09.467044 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:09.467420 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 12:34:09.467536 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 12:34:09.483283 21065 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:09.484256 21065 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:09.488040 21065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 661ff71e85ef4ae081df1c8abb8bb9fa: Generated new TSK 0
I20260619 12:34:09.488891 21065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:34:09.498308 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 20987
2026-06-19T12:34:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:09 dist-test-slave-fg3h krb5kdc[21086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:09 dist-test-slave-fg3h krb5kdc[21086](info): set up 2 sockets
Jun 19 12:34:09 dist-test-slave-fg3h krb5kdc[21086](info): commencing operation
krb5kdc: starting...
W20260619 12:34:10.545482 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:10 dist-test-slave-fg3h krb5kdc[21086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:34:10.671895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:10.784082 21099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:10.784417 21099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:10.784497 21099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:10.788609 21099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:10.788725 21099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:10.788767 21099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:10.788803 21099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:10.788838 21099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:10.794610 21099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:10.795797 21099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:10.796865 21099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:34:10.805518 21099 server_base.cc:1034] running on GCE node
W20260619 12:34:10.805457 21104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:10.805452 21105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:10.805454 21107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:10.806381 21099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:10.807585 21099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:10.808808 21099 hybrid_clock.cc:648] HybridClock initialized: now 1781872450808755 us; error 65 us; skew 500 ppm
Jun 19 12:34:10 dist-test-slave-fg3h krb5kdc[21086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:10.811939 21099 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:34:10.812809 21099 webserver.cc:460] Webserver started at http://127.0.0.1:34175/ using document root <none> and password file <none>
I20260619 12:34:10.813424 21099 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:10.813490 21099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:10.813668 21099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:10.815764 21099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "96936e936c534f0a97fea3bc83dfdff5"
format_stamp: "Formatted at 2026-06-19 12:34:10 on dist-test-slave-fg3h"
server_key: "ab9f71261aeea639b87b90e614ef9df2"
server_key_iv: "f50ac6df74999965767fe3ddbca22f4a"
server_key_version: "encryptionkey@0"
I20260619 12:34:10.816293 21099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "96936e936c534f0a97fea3bc83dfdff5"
format_stamp: "Formatted at 2026-06-19 12:34:10 on dist-test-slave-fg3h"
server_key: "ab9f71261aeea639b87b90e614ef9df2"
server_key_iv: "f50ac6df74999965767fe3ddbca22f4a"
server_key_version: "encryptionkey@0"
I20260619 12:34:10.820144 21099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 12:34:10.822784 21114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:10.824186 21099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 12:34:10.824296 21099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "96936e936c534f0a97fea3bc83dfdff5"
format_stamp: "Formatted at 2026-06-19 12:34:10 on dist-test-slave-fg3h"
server_key: "ab9f71261aeea639b87b90e614ef9df2"
server_key_iv: "f50ac6df74999965767fe3ddbca22f4a"
server_key_version: "encryptionkey@0"
I20260619 12:34:10.824389 21099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:10.855293 21099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:10.859483 21099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:10.859666 21099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:10.867239 21099 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40977
I20260619 12:34:10.867266 21166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40977 every 8 connection(s)
I20260619 12:34:10.868320 21099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:10.868726 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 21099
I20260619 12:34:10.868888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:10.869282 392 external_mini_cluster.cc:1428] Setting key 81b55b0c30c48c139251bacc3ec5b7d8
I20260619 12:34:10.871662 21167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:10.877440 21167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5: Bootstrap starting.
I20260619 12:34:10.878631 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:10.870745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40799 (local address 127.0.0.1:40977)
0619 12:34:10.871284 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:10.871299 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:10.871340 (+ 41us) server_negotiation.cc:407] Connection header received
0619 12:34:10.872156 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:10.872176 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:10.872536 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:10.872906 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:10.873359 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:10.874375 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:10.875288 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:10.875796 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:10.875974 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:10.876919 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:10.876923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:10.876935 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:10.877401 (+ 466us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:10.877457 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:10.877479 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:10.877594 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:10.877722 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 12:34:10.877929 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":147,"threads_started":1}
I20260619 12:34:10.879838 21167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:10.880636 21167 log.cc:824] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:10.881510 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:34:10.883036 21167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5: No bootstrap required, opened a new log
I20260619 12:34:10.887481 21167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96936e936c534f0a97fea3bc83dfdff5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260619 12:34:10.887813 21167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:10.887902 21167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96936e936c534f0a97fea3bc83dfdff5, State: Initialized, Role: FOLLOWER
I20260619 12:34:10.888296 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:10.883793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48809 (local address 127.0.0.1:40977)
0619 12:34:10.883931 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:10.883936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:10.884136 (+ 200us) server_negotiation.cc:407] Connection header received
0619 12:34:10.884305 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:10.884307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:10.884343 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:10.884408 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:10.884767 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:10.885372 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:10.886604 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:10.886840 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:10.886926 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:10.887785 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:10.887788 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:10.887790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:10.887800 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:10.887863 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:10.887877 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:10.887959 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:10.888088 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 12:34:10.888132 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:34:10.888453 21167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96936e936c534f0a97fea3bc83dfdff5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260619 12:34:10.888628 21167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:10.888711 21167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:10.888805 21167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:10.889849 21167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96936e936c534f0a97fea3bc83dfdff5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260619 12:34:10.890206 21167 leader_election.cc:302] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96936e936c534f0a97fea3bc83dfdff5; no voters:
I20260619 12:34:10.890516 21167 leader_election.cc:288] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:10.890586 21177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:10.890779 21177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5 [term 1 LEADER]: Becoming Leader. State: Replica: 96936e936c534f0a97fea3bc83dfdff5, State: Running, Role: LEADER
I20260619 12:34:10.891083 21177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96936e936c534f0a97fea3bc83dfdff5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260619 12:34:10.891822 21167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:10.892848 21179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96936e936c534f0a97fea3bc83dfdff5. Latest consensus state: current_term: 1 leader_uuid: "96936e936c534f0a97fea3bc83dfdff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96936e936c534f0a97fea3bc83dfdff5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } } }
I20260619 12:34:10.892988 21179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:10.893194 21178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96936e936c534f0a97fea3bc83dfdff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96936e936c534f0a97fea3bc83dfdff5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } } }
I20260619 12:34:10.893296 21178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:10.894232 21186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:10.896941 21186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:10.902818 21186 catalog_manager.cc:1347] Generated new cluster ID: b11be05a9a7942e599e18fcd76a08111
I20260619 12:34:10.902894 21186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:10.913756 21186 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:10.915086 21186 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:10.923710 21186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96936e936c534f0a97fea3bc83dfdff5: Generated new TSK 0
I20260619 12:34:10.924582 21186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:34:10.929411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 21099
2026-06-19T12:34:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:10 dist-test-slave-fg3h krb5kdc[21198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:10 dist-test-slave-fg3h krb5kdc[21198](info): set up 2 sockets
Jun 19 12:34:10 dist-test-slave-fg3h krb5kdc[21198](info): commencing operation
krb5kdc: starting...
W20260619 12:34:11.969599 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:12 dist-test-slave-fg3h krb5kdc[21198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:34:12.099749 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40727
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:12.214634 21211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:12.214918 21211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:12.214963 21211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:12.218972 21211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:12.219043 21211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:12.219062 21211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:12.219080 21211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:12.219096 21211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:12.224530 21211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:12.225613 21211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:12.226601 21211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:12.235385 21216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:12.235419 21217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:12.235385 21219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:12.235699 21211 server_base.cc:1034] running on GCE node
I20260619 12:34:12.236415 21211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:12.237640 21211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:12.238837 21211 hybrid_clock.cc:648] HybridClock initialized: now 1781872452238800 us; error 49 us; skew 500 ppm
Jun 19 12:34:12 dist-test-slave-fg3h krb5kdc[21198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:12.241777 21211 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:34:12.242729 21211 webserver.cc:460] Webserver started at http://127.0.0.1:36861/ using document root <none> and password file <none>
I20260619 12:34:12.243307 21211 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:12.243369 21211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:12.243611 21211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:12.245796 21211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "9bfb7630c8d34ed19e097c3914b11c5e"
format_stamp: "Formatted at 2026-06-19 12:34:12 on dist-test-slave-fg3h"
server_key: "6db55eb48d1784ad60b81f9ae4337427"
server_key_iv: "09c9b507388ba6b7d7bcc47d8c3f0a0b"
server_key_version: "encryptionkey@0"
I20260619 12:34:12.246411 21211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "9bfb7630c8d34ed19e097c3914b11c5e"
format_stamp: "Formatted at 2026-06-19 12:34:12 on dist-test-slave-fg3h"
server_key: "6db55eb48d1784ad60b81f9ae4337427"
server_key_iv: "09c9b507388ba6b7d7bcc47d8c3f0a0b"
server_key_version: "encryptionkey@0"
I20260619 12:34:12.250306 21211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:34:12.252904 21226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:12.254374 21211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:34:12.254544 21211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "9bfb7630c8d34ed19e097c3914b11c5e"
format_stamp: "Formatted at 2026-06-19 12:34:12 on dist-test-slave-fg3h"
server_key: "6db55eb48d1784ad60b81f9ae4337427"
server_key_iv: "09c9b507388ba6b7d7bcc47d8c3f0a0b"
server_key_version: "encryptionkey@0"
I20260619 12:34:12.254685 21211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:12.259884 21211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:12.263152 21211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:12.263361 21211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:12.271417 21211 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40727
I20260619 12:34:12.271428 21278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40727 every 8 connection(s)
I20260619 12:34:12.272459 21211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:12.275424 21279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:12.276517 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 21211
I20260619 12:34:12.276672 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:12.276979 392 external_mini_cluster.cc:1428] Setting key 479f749ea73dae874a9235b0ce195e0d
I20260619 12:34:12.281898 21279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e: Bootstrap starting.
I20260619 12:34:12.282620 21282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:12.278468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46179 (local address 127.0.0.1:40727)
0619 12:34:12.278952 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:12.278976 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:12.279014 (+ 38us) server_negotiation.cc:407] Connection header received
0619 12:34:12.279731 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:12.279750 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:12.280055 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:12.280410 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:12.281350 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:12.281355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:12.281367 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:12.281377 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:12.281435 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:12.281456 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:12.281608 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:12.281683 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 12:34:12.281852 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":152,"threads_started":1}
I20260619 12:34:12.284430 21279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:12.285343 21279 log.cc:824] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:12.285625 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:34:12.287750 21279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e: No bootstrap required, opened a new log
I20260619 12:34:12.289625 21282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:12.287915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45775 (local address 127.0.0.1:40727)
0619 12:34:12.288050 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:12.288055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:12.288213 (+ 158us) server_negotiation.cc:407] Connection header received
0619 12:34:12.288359 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:12.288361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:12.288392 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:12.288483 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:12.289224 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:12.289227 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:12.289229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:12.289239 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:12.289264 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:12.289274 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:12.289350 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:12.289448 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 12:34:12.289479 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 12:34:12.292023 21279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfb7630c8d34ed19e097c3914b11c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40727 } }
I20260619 12:34:12.292295 21279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:12.292357 21279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb7630c8d34ed19e097c3914b11c5e, State: Initialized, Role: FOLLOWER
I20260619 12:34:12.292778 21279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfb7630c8d34ed19e097c3914b11c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40727 } }
I20260619 12:34:12.292933 21279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:12.293026 21279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:12.293154 21279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:12.294149 21279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfb7630c8d34ed19e097c3914b11c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40727 } }
I20260619 12:34:12.294503 21279 leader_election.cc:302] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bfb7630c8d34ed19e097c3914b11c5e; no voters:
I20260619 12:34:12.294832 21279 leader_election.cc:288] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:12.294971 21289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:12.295238 21289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e [term 1 LEADER]: Becoming Leader. State: Replica: 9bfb7630c8d34ed19e097c3914b11c5e, State: Running, Role: LEADER
I20260619 12:34:12.295594 21289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfb7630c8d34ed19e097c3914b11c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40727 } }
I20260619 12:34:12.296211 21279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:12.297399 21291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bfb7630c8d34ed19e097c3914b11c5e. Latest consensus state: current_term: 1 leader_uuid: "9bfb7630c8d34ed19e097c3914b11c5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfb7630c8d34ed19e097c3914b11c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40727 } } }
I20260619 12:34:12.297534 21291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:12.297847 21290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bfb7630c8d34ed19e097c3914b11c5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfb7630c8d34ed19e097c3914b11c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40727 } } }
I20260619 12:34:12.297960 21290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:12.297940 21298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:12.301389 21298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:12.307523 21298 catalog_manager.cc:1347] Generated new cluster ID: 52d805de72b84eb487cc99514097f83e
I20260619 12:34:12.307603 21298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:12.330950 21298 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:12.331321 21298 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:12.344136 21298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bfb7630c8d34ed19e097c3914b11c5e: Generated new TSK 0
I20260619 12:34:12.344930 21298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:34:12.360510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 21211
2026-06-19T12:34:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:12 dist-test-slave-fg3h krb5kdc[21310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:12 dist-test-slave-fg3h krb5kdc[21310](info): set up 2 sockets
Jun 19 12:34:12 dist-test-slave-fg3h krb5kdc[21310](info): commencing operation
krb5kdc: starting...
W20260619 12:34:13.409358 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:13 dist-test-slave-fg3h krb5kdc[21310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:34:13.532418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:13.647042 21323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:13.647353 21323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:13.647441 21323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:13.651705 21323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:13.651804 21323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:13.651844 21323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:13.651881 21323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:13.651919 21323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:13.657658 21323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:13.658819 21323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:13.659891 21323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:13.668853 21328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:13.668849 21331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:13.668907 21323 server_base.cc:1034] running on GCE node
W20260619 12:34:13.668849 21329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:13.669718 21323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:13.670836 21323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:13.672040 21323 hybrid_clock.cc:648] HybridClock initialized: now 1781872453672005 us; error 47 us; skew 500 ppm
Jun 19 12:34:13 dist-test-slave-fg3h krb5kdc[21310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:13.674975 21323 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:34:13.675875 21323 webserver.cc:460] Webserver started at http://127.0.0.1:45741/ using document root <none> and password file <none>
I20260619 12:34:13.676442 21323 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:13.676533 21323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:13.676765 21323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:13.678848 21323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "074a88171a3e4ecd8dca42bb8c629f55"
format_stamp: "Formatted at 2026-06-19 12:34:13 on dist-test-slave-fg3h"
server_key: "136314ac388560ceab5e28ab441036c2"
server_key_iv: "56064e9bef5b7e0dcaf22fdb709373ca"
server_key_version: "encryptionkey@0"
I20260619 12:34:13.679415 21323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "074a88171a3e4ecd8dca42bb8c629f55"
format_stamp: "Formatted at 2026-06-19 12:34:13 on dist-test-slave-fg3h"
server_key: "136314ac388560ceab5e28ab441036c2"
server_key_iv: "56064e9bef5b7e0dcaf22fdb709373ca"
server_key_version: "encryptionkey@0"
I20260619 12:34:13.683183 21323 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 12:34:13.685698 21338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:13.686936 21323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:34:13.687086 21323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "074a88171a3e4ecd8dca42bb8c629f55"
format_stamp: "Formatted at 2026-06-19 12:34:13 on dist-test-slave-fg3h"
server_key: "136314ac388560ceab5e28ab441036c2"
server_key_iv: "56064e9bef5b7e0dcaf22fdb709373ca"
server_key_version: "encryptionkey@0"
I20260619 12:34:13.687207 21323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:13.699055 21323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:13.702196 21323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:13.702402 21323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:13.709833 21323 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39775
I20260619 12:34:13.709913 21390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39775 every 8 connection(s)
I20260619 12:34:13.710789 21323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:13.713858 21391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:13.719084 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 21323
I20260619 12:34:13.719254 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:13.719548 392 external_mini_cluster.cc:1428] Setting key 39493e8612af4ae4817402816e3a1ce8
I20260619 12:34:13.720103 21391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55: Bootstrap starting.
I20260619 12:34:13.723197 21391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:13.724141 21391 log.cc:824] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:13.726375 21391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55: No bootstrap required, opened a new log
Jun 19 12:34:13 dist-test-slave-fg3h krb5kdc[21310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 12:34:13.730988 21391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074a88171a3e4ecd8dca42bb8c629f55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39775 } }
I20260619 12:34:13.731338 21391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:13.731410 21391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a88171a3e4ecd8dca42bb8c629f55, State: Initialized, Role: FOLLOWER
I20260619 12:34:13.731938 21391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074a88171a3e4ecd8dca42bb8c629f55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39775 } }
I20260619 12:34:13.732115 21391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:13.732218 21391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:13.732324 21391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:13.733387 21391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074a88171a3e4ecd8dca42bb8c629f55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39775 } }
I20260619 12:34:13.733755 21391 leader_election.cc:302] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 074a88171a3e4ecd8dca42bb8c629f55; no voters:
I20260619 12:34:13.734157 21391 leader_election.cc:288] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:13.734208 21396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:13.734483 21396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55 [term 1 LEADER]: Becoming Leader. State: Replica: 074a88171a3e4ecd8dca42bb8c629f55, State: Running, Role: LEADER
I20260619 12:34:13.734848 21396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074a88171a3e4ecd8dca42bb8c629f55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39775 } }
I20260619 12:34:13.735570 21391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:13.735790 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:13.720915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:49639 (local address 127.0.0.1:39775)
0619 12:34:13.721414 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:13.721438 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:13.721501 (+ 63us) server_negotiation.cc:407] Connection header received
0619 12:34:13.722411 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:13.722435 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:13.722912 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:13.723310 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:13.724024 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:13.725615 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:13.726493 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:13.727222 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:13.727398 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:13.730250 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:13.730256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:13.730275 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:13.730310 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:13.732314 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:13.732831 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:13.732836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:13.732842 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:13.732961 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:13.733378 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:13.733381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:13.733383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:13.733836 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:13.734050 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:13.734516 (+ 466us) server_negotiation.cc:299] Negotiation successful
0619 12:34:13.734763 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":180,"threads_started":1}
I20260619 12:34:13.736510 21397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "074a88171a3e4ecd8dca42bb8c629f55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074a88171a3e4ecd8dca42bb8c629f55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39775 } } }
I20260619 12:34:13.736654 21397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:13.737030 21398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 074a88171a3e4ecd8dca42bb8c629f55. Latest consensus state: current_term: 1 leader_uuid: "074a88171a3e4ecd8dca42bb8c629f55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074a88171a3e4ecd8dca42bb8c629f55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39775 } } }
I20260619 12:34:13.737215 21398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:13.737510 21405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:13.739606 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:34:13.742302 21405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:13.748826 21405 catalog_manager.cc:1347] Generated new cluster ID: 30871832f28449e8bbf26e6e78344f51
I20260619 12:34:13.748989 21405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:13.754544 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:13.745583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:44481 (local address 127.0.0.1:39775)
0619 12:34:13.745719 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:13.745723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:13.745954 (+ 231us) server_negotiation.cc:407] Connection header received
0619 12:34:13.746104 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:13.746107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:13.746159 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:13.746243 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:13.747012 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:13.748260 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:13.749367 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:13.749612 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:13.749733 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:13.751159 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:13.751163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:13.751165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:13.751189 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:13.752717 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:13.753232 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:13.753236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:13.753237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:13.753317 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:13.753676 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:13.753680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:13.753682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:13.753852 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:13.754006 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:13.754275 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 12:34:13.754368 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 12:34:13.762437 21405 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:13.763465 21405 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:13.771520 21405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 074a88171a3e4ecd8dca42bb8c629f55: Generated new TSK 0
I20260619 12:34:13.772272 21405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:34:13.776355 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 21323
W20260619 12:34:13.782176 21308 connection.cc:535] client connection to 127.0.0.1:39775 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39775): Connection reset by peer (error 104)
2026-06-19T12:34:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:13 dist-test-slave-fg3h krb5kdc[21422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:13 dist-test-slave-fg3h krb5kdc[21422](info): set up 2 sockets
Jun 19 12:34:13 dist-test-slave-fg3h krb5kdc[21422](info): commencing operation
krb5kdc: starting...
W20260619 12:34:14.821383 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:14 dist-test-slave-fg3h krb5kdc[21422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:34:14.945674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45809
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:15.062749 21435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:15.063035 21435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:15.063128 21435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:15.067301 21435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:15.067368 21435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:15.067386 21435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:15.067404 21435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:15.067444 21435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:15.072872 21435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:15.073971 21435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:15.075017 21435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:34:15.083798 21435 server_base.cc:1034] running on GCE node
W20260619 12:34:15.083747 21440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:15.083792 21443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:15.083747 21441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:15.084499 21435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:15.085693 21435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:15.086864 21435 hybrid_clock.cc:648] HybridClock initialized: now 1781872455086847 us; error 28 us; skew 500 ppm
Jun 19 12:34:15 dist-test-slave-fg3h krb5kdc[21422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:15.093677 21435 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:34:15.094630 21435 webserver.cc:460] Webserver started at http://127.0.0.1:45551/ using document root <none> and password file <none>
I20260619 12:34:15.095243 21435 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:15.095337 21435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:15.095571 21435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:15.097765 21435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "d21242dba65c4110914ad3c21c5d8ed0"
format_stamp: "Formatted at 2026-06-19 12:34:15 on dist-test-slave-fg3h"
server_key: "559aaeabf140d6b6e53610b4cd24f018"
server_key_iv: "f308bbb2be0c7d81e9b3b8c25e014fee"
server_key_version: "encryptionkey@0"
I20260619 12:34:15.098351 21435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "d21242dba65c4110914ad3c21c5d8ed0"
format_stamp: "Formatted at 2026-06-19 12:34:15 on dist-test-slave-fg3h"
server_key: "559aaeabf140d6b6e53610b4cd24f018"
server_key_iv: "f308bbb2be0c7d81e9b3b8c25e014fee"
server_key_version: "encryptionkey@0"
I20260619 12:34:15.103304 21435 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 12:34:15.109010 21450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:15.111234 21435 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260619 12:34:15.111398 21435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "d21242dba65c4110914ad3c21c5d8ed0"
format_stamp: "Formatted at 2026-06-19 12:34:15 on dist-test-slave-fg3h"
server_key: "559aaeabf140d6b6e53610b4cd24f018"
server_key_iv: "f308bbb2be0c7d81e9b3b8c25e014fee"
server_key_version: "encryptionkey@0"
I20260619 12:34:15.111559 21435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:15.133744 21435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:15.138510 21435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:15.138738 21435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:15.148543 21435 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45809
I20260619 12:34:15.148573 21502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45809 every 8 connection(s)
I20260619 12:34:15.149725 21435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:15.152446 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 21435
I20260619 12:34:15.152616 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:15.152944 392 external_mini_cluster.cc:1428] Setting key 7fb08481db6afc9ccf1c3a9ee70eda32
I20260619 12:34:15.153111 21503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:15.160207 21503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0: Bootstrap starting.
I20260619 12:34:15.162523 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:15.154553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:54131 (local address 127.0.0.1:45809)
0619 12:34:15.155028 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:15.155045 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:15.155081 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:34:15.156148 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:15.156166 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:15.156524 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:15.156864 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:15.157294 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:15.158288 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:15.159200 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:15.159678 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:15.159851 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:15.160842 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:15.160847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:15.160860 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:15.161312 (+ 452us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:15.161366 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:15.161388 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:15.161534 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:15.161673 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 12:34:15.161813 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":152,"threads_started":1}
I20260619 12:34:15.162757 21503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:15.163556 21503 log.cc:824] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:15.165745 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:34:15.165915 21503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0: No bootstrap required, opened a new log
I20260619 12:34:15.170168 21503 raft_consensus.cc:348] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21242dba65c4110914ad3c21c5d8ed0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45809 } }
I20260619 12:34:15.170450 21503 raft_consensus.cc:374] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:15.170514 21503 raft_consensus.cc:729] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d21242dba65c4110914ad3c21c5d8ed0, State: Initialized, Role: FOLLOWER
I20260619 12:34:15.170997 21503 consensus_queue.cc:260] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21242dba65c4110914ad3c21c5d8ed0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45809 } }
I20260619 12:34:15.171166 21503 raft_consensus.cc:388] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:15.171236 21503 raft_consensus.cc:482] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:15.171340 21503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:15.172361 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:15.167889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:34611 (local address 127.0.0.1:45809)
0619 12:34:15.168054 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:15.168060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:15.168160 (+ 100us) server_negotiation.cc:407] Connection header received
0619 12:34:15.168241 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:15.168244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:15.168289 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:15.168375 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:15.168600 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:15.169385 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:15.170383 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:15.170718 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:15.170827 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:15.171759 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:15.171762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:15.171764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:15.171775 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:15.171798 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:15.171808 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:15.171879 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:15.172166 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 12:34:15.172204 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 12:34:15.172573 21503 raft_consensus.cc:504] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21242dba65c4110914ad3c21c5d8ed0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45809 } }
I20260619 12:34:15.172933 21503 leader_election.cc:302] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d21242dba65c4110914ad3c21c5d8ed0; no voters:
I20260619 12:34:15.173295 21503 leader_election.cc:288] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:15.173424 21513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:15.173691 21513 raft_consensus.cc:686] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0 [term 1 LEADER]: Becoming Leader. State: Replica: d21242dba65c4110914ad3c21c5d8ed0, State: Running, Role: LEADER
I20260619 12:34:15.174021 21513 consensus_queue.cc:237] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21242dba65c4110914ad3c21c5d8ed0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45809 } }
I20260619 12:34:15.174597 21503 sys_catalog.cc:564] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:15.175899 21514 sys_catalog.cc:455] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d21242dba65c4110914ad3c21c5d8ed0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21242dba65c4110914ad3c21c5d8ed0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45809 } } }
I20260619 12:34:15.175905 21515 sys_catalog.cc:455] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d21242dba65c4110914ad3c21c5d8ed0. Latest consensus state: current_term: 1 leader_uuid: "d21242dba65c4110914ad3c21c5d8ed0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21242dba65c4110914ad3c21c5d8ed0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45809 } } }
I20260619 12:34:15.176041 21515 sys_catalog.cc:458] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:15.176040 21514 sys_catalog.cc:458] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0 [sys.catalog]: This master's current role is: LEADER
W20260619 12:34:15.179550 21528 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:34:15.179620 21528 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:34:15.179704 21523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:15.180430 21523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:15.186175 21523 catalog_manager.cc:1347] Generated new cluster ID: 5552f45ff32149e6ad54732d67aaf821
I20260619 12:34:15.186244 21523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:15.195073 21523 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:15.196370 21523 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:15.205343 21523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d21242dba65c4110914ad3c21c5d8ed0: Generated new TSK 0
I20260619 12:34:15.206130 21523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:34:15.217048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 21435
W20260619 12:34:15.222577 21420 connection.cc:535] client connection to 127.0.0.1:45809 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45809): Connection reset by peer (error 104)
2026-06-19T12:34:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:15 dist-test-slave-fg3h krb5kdc[21534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:15 dist-test-slave-fg3h krb5kdc[21534](info): set up 2 sockets
Jun 19 12:34:15 dist-test-slave-fg3h krb5kdc[21534](info): commencing operation
krb5kdc: starting...
W20260619 12:34:16.305522 392 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:16 dist-test-slave-fg3h krb5kdc[21534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:34:16.430406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33175
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:16.540772 21547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:16.541065 21547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:16.541205 21547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:16.545254 21547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:16.545317 21547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:16.545337 21547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:16.545354 21547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:16.545369 21547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:16.551061 21547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:16.552131 21547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:16.553166 21547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:16.561981 21555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:16.562079 21553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:16.562275 21552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:16.562157 21547 server_base.cc:1034] running on GCE node
I20260619 12:34:16.562928 21547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:16.564025 21547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:16.565178 21547 hybrid_clock.cc:648] HybridClock initialized: now 1781872456565157 us; error 32 us; skew 500 ppm
Jun 19 12:34:16 dist-test-slave-fg3h krb5kdc[21534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:16.568210 21547 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:34:16.569267 21547 webserver.cc:460] Webserver started at http://127.0.0.1:41203/ using document root <none> and password file <none>
I20260619 12:34:16.569842 21547 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:16.569903 21547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:16.570174 21547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:16.572201 21547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "41c132cfbb5b4362b5b8717cde3282be"
format_stamp: "Formatted at 2026-06-19 12:34:16 on dist-test-slave-fg3h"
server_key: "cb20e948e793161445c9b4582685b36c"
server_key_iv: "8892bc29943c097555c9c4b7f33d3b1f"
server_key_version: "encryptionkey@0"
I20260619 12:34:16.572762 21547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "41c132cfbb5b4362b5b8717cde3282be"
format_stamp: "Formatted at 2026-06-19 12:34:16 on dist-test-slave-fg3h"
server_key: "cb20e948e793161445c9b4582685b36c"
server_key_iv: "8892bc29943c097555c9c4b7f33d3b1f"
server_key_version: "encryptionkey@0"
I20260619 12:34:16.576536 21547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:34:16.579101 21562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:16.580456 21547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:34:16.580603 21547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "41c132cfbb5b4362b5b8717cde3282be"
format_stamp: "Formatted at 2026-06-19 12:34:16 on dist-test-slave-fg3h"
server_key: "cb20e948e793161445c9b4582685b36c"
server_key_iv: "8892bc29943c097555c9c4b7f33d3b1f"
server_key_version: "encryptionkey@0"
I20260619 12:34:16.580724 21547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:16.585467 21547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:16.588821 21547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:16.589030 21547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:16.596350 21547 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33175
I20260619 12:34:16.596369 21614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33175 every 8 connection(s)
I20260619 12:34:16.597352 21547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:16.600101 21615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:16.605507 21615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be: Bootstrap starting.
I20260619 12:34:16.606796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 21547
I20260619 12:34:16.606985 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:16.607281 392 external_mini_cluster.cc:1428] Setting key e10ac362cdb93c3e6fe39e720caf9946
I20260619 12:34:16.608098 21615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:16.608968 21615 log.cc:824] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:16.611048 21615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be: No bootstrap required, opened a new log
I20260619 12:34:16.612201 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:16.608387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:59415 (local address 127.0.0.1:33175)
0619 12:34:16.608800 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:16.608813 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:16.608847 (+ 34us) server_negotiation.cc:407] Connection header received
0619 12:34:16.609552 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:16.609570 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:16.609839 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:16.610124 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:16.610993 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:16.610997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:16.611008 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:16.611018 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:16.611075 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:16.611094 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:16.611225 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:16.611383 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 12:34:16.611544 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":136,"threads_started":1}
I20260619 12:34:16.614985 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:34:16.615236 21615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c132cfbb5b4362b5b8717cde3282be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33175 } }
I20260619 12:34:16.615458 21615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:16.615541 21615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c132cfbb5b4362b5b8717cde3282be, State: Initialized, Role: FOLLOWER
I20260619 12:34:16.615942 21615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c132cfbb5b4362b5b8717cde3282be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33175 } }
I20260619 12:34:16.616086 21615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:16.616174 21615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:16.616290 21615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:16.617372 21615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c132cfbb5b4362b5b8717cde3282be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33175 } }
I20260619 12:34:16.617748 21615 leader_election.cc:302] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41c132cfbb5b4362b5b8717cde3282be; no voters:
I20260619 12:34:16.618095 21615 leader_election.cc:288] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:16.618165 21625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:16.618350 21625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be [term 1 LEADER]: Becoming Leader. State: Replica: 41c132cfbb5b4362b5b8717cde3282be, State: Running, Role: LEADER
I20260619 12:34:16.618680 21625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c132cfbb5b4362b5b8717cde3282be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33175 } }
I20260619 12:34:16.618939 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:16.617152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:41041 (local address 127.0.0.1:33175)
0619 12:34:16.617261 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:16.617265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:16.617421 (+ 156us) server_negotiation.cc:407] Connection header received
0619 12:34:16.617553 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:16.617558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:16.617593 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:16.617691 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:16.618513 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:16.618516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:16.618518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:16.618529 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:16.618553 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:16.618564 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:16.618620 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:16.618797 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 12:34:16.618836 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 12:34:16.619354 21615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:16.620373 21627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41c132cfbb5b4362b5b8717cde3282be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c132cfbb5b4362b5b8717cde3282be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33175 } } }
I20260619 12:34:16.620635 21626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41c132cfbb5b4362b5b8717cde3282be. Latest consensus state: current_term: 1 leader_uuid: "41c132cfbb5b4362b5b8717cde3282be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c132cfbb5b4362b5b8717cde3282be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33175 } } }
I20260619 12:34:16.620750 21627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:16.620815 21626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be [sys.catalog]: This master's current role is: LEADER
W20260619 12:34:16.623953 21640 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:34:16.624020 21640 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:34:16.624110 21636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:16.624806 21636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:16.630182 21636 catalog_manager.cc:1347] Generated new cluster ID: ae34d8bcf8d04dd7a57867d0d35ef065
I20260619 12:34:16.630260 21636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:16.650750 21636 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:16.651077 21636 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:16.655920 21636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41c132cfbb5b4362b5b8717cde3282be: Generated new TSK 0
I20260619 12:34:16.656555 21636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:34:16.667552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 21547
2026-06-19T12:34:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1449 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:16 dist-test-slave-fg3h krb5kdc[21646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:16 dist-test-slave-fg3h krb5kdc[21646](info): set up 2 sockets
Jun 19 12:34:16 dist-test-slave-fg3h krb5kdc[21646](info): commencing operation
krb5kdc: starting...
W20260619 12:34:17.705852 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:17 dist-test-slave-fg3h krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:34:17.826305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38069
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:17.936307 21659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:17.936617 21659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:17.936702 21659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:17.940757 21659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:17.940860 21659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:17.940899 21659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:17.940960 21659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:17.941015 21659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:17.946333 21659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38069
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:17.947381 21659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:17.948408 21659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:17.956634 21665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:17.956792 21659 server_base.cc:1034] running on GCE node
W20260619 12:34:17.956634 21664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:17.956640 21667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:17.957629 21659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:17.958743 21659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:17.959955 21659 hybrid_clock.cc:648] HybridClock initialized: now 1781872457959939 us; error 35 us; skew 500 ppm
Jun 19 12:34:17 dist-test-slave-fg3h krb5kdc[21646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:17.962746 21659 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:34:17.963651 21659 webserver.cc:460] Webserver started at http://127.0.0.1:46737/ using document root <none> and password file <none>
I20260619 12:34:17.964193 21659 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:17.964253 21659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:17.964455 21659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:17.966542 21659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "338260cf71aa43f1abfe881db6de112d"
format_stamp: "Formatted at 2026-06-19 12:34:17 on dist-test-slave-fg3h"
server_key: "6cd22ae60a497981fd7f3511287260d6"
server_key_iv: "60a8e84ac738eafe460ace1cecf4bb0a"
server_key_version: "encryptionkey@0"
I20260619 12:34:17.967109 21659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "338260cf71aa43f1abfe881db6de112d"
format_stamp: "Formatted at 2026-06-19 12:34:17 on dist-test-slave-fg3h"
server_key: "6cd22ae60a497981fd7f3511287260d6"
server_key_iv: "60a8e84ac738eafe460ace1cecf4bb0a"
server_key_version: "encryptionkey@0"
I20260619 12:34:17.970821 21659 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 12:34:17.973263 21674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:17.974483 21659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:34:17.974637 21659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "338260cf71aa43f1abfe881db6de112d"
format_stamp: "Formatted at 2026-06-19 12:34:17 on dist-test-slave-fg3h"
server_key: "6cd22ae60a497981fd7f3511287260d6"
server_key_iv: "60a8e84ac738eafe460ace1cecf4bb0a"
server_key_version: "encryptionkey@0"
I20260619 12:34:17.974759 21659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:17.984242 21659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:17.987169 21659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:17.987371 21659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:17.994377 21659 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38069
I20260619 12:34:17.994400 21726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38069 every 8 connection(s)
I20260619 12:34:17.995288 21659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:17.998211 21727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:18.002434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 21659
I20260619 12:34:18.002586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:18.002915 392 external_mini_cluster.cc:1428] Setting key 46f800cc206353abd7551f3b02584afc
I20260619 12:34:18.005622 21727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d: Bootstrap starting.
I20260619 12:34:18.008224 21727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:18.009136 21727 log.cc:824] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:18.011670 21727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d: No bootstrap required, opened a new log
Jun 19 12:34:18 dist-test-slave-fg3h krb5kdc[21646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 12:34:18.016027 21727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338260cf71aa43f1abfe881db6de112d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38069 } }
I20260619 12:34:18.016258 21727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:18.016336 21727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 338260cf71aa43f1abfe881db6de112d, State: Initialized, Role: FOLLOWER
I20260619 12:34:18.016840 21727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338260cf71aa43f1abfe881db6de112d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38069 } }
I20260619 12:34:18.017010 21727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:18.017122 21727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:18.017253 21727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:18.018298 21727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338260cf71aa43f1abfe881db6de112d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38069 } }
I20260619 12:34:18.018411 21730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:18.004628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:37419 (local address 127.0.0.1:38069)
0619 12:34:18.005156 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:18.005183 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:18.005219 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:34:18.005909 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:18.005929 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:18.006266 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:18.006545 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:18.007479 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:18.008594 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:18.009821 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:18.010439 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:18.010565 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:18.013504 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:18.013509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:18.013523 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:18.013548 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:18.015368 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:18.015893 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:18.015898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:18.015903 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:18.016011 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:18.016393 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:18.016398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:18.016405 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:18.016864 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:18.017095 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:18.017451 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 12:34:18.017674 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":142,"threads_started":1}
I20260619 12:34:18.018733 21727 leader_election.cc:302] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 338260cf71aa43f1abfe881db6de112d; no voters:
I20260619 12:34:18.019083 21727 leader_election.cc:288] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:18.019200 21732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:18.019436 21732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d [term 1 LEADER]: Becoming Leader. State: Replica: 338260cf71aa43f1abfe881db6de112d, State: Running, Role: LEADER
I20260619 12:34:18.019775 21732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338260cf71aa43f1abfe881db6de112d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38069 } }
I20260619 12:34:18.020344 21727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:18.021399 21733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "338260cf71aa43f1abfe881db6de112d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338260cf71aa43f1abfe881db6de112d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38069 } } }
I20260619 12:34:18.021524 21733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:18.021685 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:34:18.021874 21734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 338260cf71aa43f1abfe881db6de112d. Latest consensus state: current_term: 1 leader_uuid: "338260cf71aa43f1abfe881db6de112d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "338260cf71aa43f1abfe881db6de112d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38069 } } }
I20260619 12:34:18.021987 21734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d [sys.catalog]: This master's current role is: LEADER
W20260619 12:34:18.026227 21747 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:34:18.026361 21747 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:34:18.026476 21741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:18.027364 21741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:18.034053 21741 catalog_manager.cc:1347] Generated new cluster ID: 5084486be74849dd81bc9122c4d2db63
I20260619 12:34:18.034135 21741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:18.040334 21730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:18.023946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:43559 (local address 127.0.0.1:38069)
0619 12:34:18.024072 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:18.024076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:18.024354 (+ 278us) server_negotiation.cc:407] Connection header received
0619 12:34:18.024405 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:18.024409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:18.024486 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:18.024572 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:18.025444 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:18.026233 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:18.028377 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:18.028644 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:18.028760 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:18.030681 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:18.030685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:18.030688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:18.030714 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:18.038287 (+ 7573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:18.038874 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:18.038878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:18.038880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:18.038968 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:18.039302 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:18.039305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:18.039307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:18.039523 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:18.039676 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:18.040079 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 12:34:18.040182 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 12:34:18.094233 21741 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:18.095245 21741 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:18.102854 21741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 338260cf71aa43f1abfe881db6de112d: Generated new TSK 0
I20260619 12:34:18.103658 21741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:34:18.111608 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 21659
W20260619 12:34:18.116837 21644 connection.cc:535] client connection to 127.0.0.1:38069 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38069): Connection reset by peer (error 104)
2026-06-19T12:34:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:18 dist-test-slave-fg3h krb5kdc[21758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:18 dist-test-slave-fg3h krb5kdc[21758](info): set up 2 sockets
Jun 19 12:34:18 dist-test-slave-fg3h krb5kdc[21758](info): commencing operation
krb5kdc: starting...
W20260619 12:34:19.145433 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:19 dist-test-slave-fg3h krb5kdc[21758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:34:19.268211 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40717
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:19.380450 21771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:19.380734 21771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:19.380776 21771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:19.384740 21771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:19.384805 21771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:19.384824 21771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:19.384841 21771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:19.384856 21771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:19.390161 21771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40717
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:19.391185 21771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:19.392146 21771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:19.400630 21776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:19.400642 21777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:19.400647 21779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:19.401038 21771 server_base.cc:1034] running on GCE node
I20260619 12:34:19.401662 21771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:19.402742 21771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:19.403913 21771 hybrid_clock.cc:648] HybridClock initialized: now 1781872459403897 us; error 29 us; skew 500 ppm
Jun 19 12:34:19 dist-test-slave-fg3h krb5kdc[21758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:19.406664 21771 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:34:19.407557 21771 webserver.cc:460] Webserver started at http://127.0.0.1:38561/ using document root <none> and password file <none>
I20260619 12:34:19.408087 21771 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:19.408143 21771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:19.408303 21771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:19.410362 21771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "61d65738a1c346a5bc45d31fe15fbef4"
format_stamp: "Formatted at 2026-06-19 12:34:19 on dist-test-slave-fg3h"
server_key: "f82ba6dfb567dfc77e0fa536b33ca86d"
server_key_iv: "c5f4ed5ba512d2cee3ea1f414a7ac7c0"
server_key_version: "encryptionkey@0"
I20260619 12:34:19.410883 21771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "61d65738a1c346a5bc45d31fe15fbef4"
format_stamp: "Formatted at 2026-06-19 12:34:19 on dist-test-slave-fg3h"
server_key: "f82ba6dfb567dfc77e0fa536b33ca86d"
server_key_iv: "c5f4ed5ba512d2cee3ea1f414a7ac7c0"
server_key_version: "encryptionkey@0"
I20260619 12:34:19.414695 21771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:34:19.417238 21786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:19.418725 21771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:34:19.418865 21771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "61d65738a1c346a5bc45d31fe15fbef4"
format_stamp: "Formatted at 2026-06-19 12:34:19 on dist-test-slave-fg3h"
server_key: "f82ba6dfb567dfc77e0fa536b33ca86d"
server_key_iv: "c5f4ed5ba512d2cee3ea1f414a7ac7c0"
server_key_version: "encryptionkey@0"
I20260619 12:34:19.418994 21771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:19.439849 21771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:19.442863 21771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:19.443075 21771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:19.450636 21771 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40717
I20260619 12:34:19.450656 21838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40717 every 8 connection(s)
I20260619 12:34:19.451596 21771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:19.454488 21839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:19.454627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 21771
I20260619 12:34:19.454762 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:19.455062 392 external_mini_cluster.cc:1428] Setting key d2018cf59f4df5ed54258f1c99168247
I20260619 12:34:19.461045 21839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4: Bootstrap starting.
I20260619 12:34:19.463459 21839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:19.464316 21839 log.cc:824] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:19.464550 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:19.456488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:50189 (local address 127.0.0.1:40717)
0619 12:34:19.456964 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:19.456980 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:19.457018 (+ 38us) server_negotiation.cc:407] Connection header received
0619 12:34:19.457895 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:19.457915 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:19.458293 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:19.458669 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:19.459112 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:19.460088 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:19.461047 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:19.461630 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:19.461800 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:19.462778 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:19.462783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:19.462796 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:19.463224 (+ 428us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:19.463277 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:19.463300 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:19.463452 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:19.463630 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 12:34:19.463768 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":146,"threads_started":1}
I20260619 12:34:19.466578 21839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4: No bootstrap required, opened a new log
I20260619 12:34:19.467692 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:34:19.471092 21839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61d65738a1c346a5bc45d31fe15fbef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40717 } }
I20260619 12:34:19.471385 21839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:19.471467 21839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d65738a1c346a5bc45d31fe15fbef4, State: Initialized, Role: FOLLOWER
I20260619 12:34:19.471942 21839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61d65738a1c346a5bc45d31fe15fbef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40717 } }
I20260619 12:34:19.472100 21839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:19.472167 21839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:19.472271 21839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:19.473399 21839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61d65738a1c346a5bc45d31fe15fbef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40717 } }
I20260619 12:34:19.473775 21839 leader_election.cc:302] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61d65738a1c346a5bc45d31fe15fbef4; no voters:
I20260619 12:34:19.474097 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:19.469762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:32877 (local address 127.0.0.1:40717)
0619 12:34:19.469868 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:19.469873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:19.470110 (+ 237us) server_negotiation.cc:407] Connection header received
0619 12:34:19.470196 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:19.470198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:19.470246 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:19.470333 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:19.470592 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:19.471335 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:19.472267 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:19.472593 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:19.472722 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:19.473608 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:19.473612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:19.473615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:19.473627 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:19.473659 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:19.473670 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:19.473768 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:19.473911 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 12:34:19.473948 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":31}
I20260619 12:34:19.474167 21839 leader_election.cc:288] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:19.474270 21849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:19.474512 21849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4 [term 1 LEADER]: Becoming Leader. State: Replica: 61d65738a1c346a5bc45d31fe15fbef4, State: Running, Role: LEADER
I20260619 12:34:19.474879 21849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61d65738a1c346a5bc45d31fe15fbef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40717 } }
I20260619 12:34:19.475486 21839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:19.476074 21851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61d65738a1c346a5bc45d31fe15fbef4. Latest consensus state: current_term: 1 leader_uuid: "61d65738a1c346a5bc45d31fe15fbef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61d65738a1c346a5bc45d31fe15fbef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40717 } } }
I20260619 12:34:19.476275 21851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:19.476534 21850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61d65738a1c346a5bc45d31fe15fbef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61d65738a1c346a5bc45d31fe15fbef4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40717 } } }
I20260619 12:34:19.476626 21850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:19.476778 21858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:19.480198 21858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:19.485953 21858 catalog_manager.cc:1347] Generated new cluster ID: 72badfe44fac44ada2523a187db1c487
I20260619 12:34:19.486027 21858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:19.498056 21858 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:19.499295 21858 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:19.506595 21858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61d65738a1c346a5bc45d31fe15fbef4: Generated new TSK 0
I20260619 12:34:19.507339 21858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:34:19.519197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 21771
W20260619 12:34:19.524755 21756 connection.cc:535] client connection to 127.0.0.1:40717 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40717): Connection reset by peer (error 104)
2026-06-19T12:34:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:19 dist-test-slave-fg3h krb5kdc[21870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:19 dist-test-slave-fg3h krb5kdc[21870](info): set up 2 sockets
Jun 19 12:34:19 dist-test-slave-fg3h krb5kdc[21870](info): commencing operation
krb5kdc: starting...
W20260619 12:34:20.561448 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:20 dist-test-slave-fg3h krb5kdc[21870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 12:34:20.684053 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43911
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:20.795403 21883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:20.795733 21883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:20.795820 21883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:20.799949 21883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:20.800057 21883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:20.800096 21883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:20.800132 21883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:20.800163 21883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:20.806289 21883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43911
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:20.807390 21883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:20.808444 21883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:20.817265 21889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:20.817322 21883 server_base.cc:1034] running on GCE node
W20260619 12:34:20.817278 21891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:20.817265 21888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:20.818137 21883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:20.819221 21883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:20.820400 21883 hybrid_clock.cc:648] HybridClock initialized: now 1781872460820377 us; error 54 us; skew 500 ppm
Jun 19 12:34:20 dist-test-slave-fg3h krb5kdc[21870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:20.826655 21883 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 12:34:20.827616 21883 webserver.cc:460] Webserver started at http://127.0.0.1:43151/ using document root <none> and password file <none>
I20260619 12:34:20.828223 21883 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:20.828318 21883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:20.828563 21883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:20.830775 21883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "e864300082b9413bbad78387420d876c"
format_stamp: "Formatted at 2026-06-19 12:34:20 on dist-test-slave-fg3h"
server_key: "0a2d86951abe77e84d13ad154b92e854"
server_key_iv: "7907ec08050cab64e8697505467a41ff"
server_key_version: "encryptionkey@0"
I20260619 12:34:20.831388 21883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "e864300082b9413bbad78387420d876c"
format_stamp: "Formatted at 2026-06-19 12:34:20 on dist-test-slave-fg3h"
server_key: "0a2d86951abe77e84d13ad154b92e854"
server_key_iv: "7907ec08050cab64e8697505467a41ff"
server_key_version: "encryptionkey@0"
I20260619 12:34:20.835351 21883 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:34:20.838016 21898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:20.839268 21883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 12:34:20.839418 21883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "e864300082b9413bbad78387420d876c"
format_stamp: "Formatted at 2026-06-19 12:34:20 on dist-test-slave-fg3h"
server_key: "0a2d86951abe77e84d13ad154b92e854"
server_key_iv: "7907ec08050cab64e8697505467a41ff"
server_key_version: "encryptionkey@0"
I20260619 12:34:20.839550 21883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:20.844156 21883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:20.847707 21883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:20.847952 21883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:20.855496 21883 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43911
I20260619 12:34:20.855500 21950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43911 every 8 connection(s)
I20260619 12:34:20.856467 21883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:20.859434 21951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:20.860332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 21883
I20260619 12:34:20.860476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:20.860800 392 external_mini_cluster.cc:1428] Setting key 2007acbf30945dc26739873f61b8c27e
I20260619 12:34:20.866210 21951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c: Bootstrap starting.
I20260619 12:34:20.866334 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:20.862293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:40017 (local address 127.0.0.1:43911)
0619 12:34:20.862744 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:20.862759 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:20.862794 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:34:20.863512 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:20.863530 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:20.863825 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:20.864193 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:20.865042 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:20.865046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:20.865058 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:20.865068 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:20.865187 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:20.865211 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:20.865364 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:20.865436 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 12:34:20.865606 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":139,"threads_started":1}
I20260619 12:34:20.868793 21951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:20.869340 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:34:20.869679 21951 log.cc:824] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:20.871834 21951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c: No bootstrap required, opened a new log
I20260619 12:34:20.873404 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:20.871652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:35353 (local address 127.0.0.1:43911)
0619 12:34:20.871775 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:20.871779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:20.871954 (+ 175us) server_negotiation.cc:407] Connection header received
0619 12:34:20.872071 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:20.872074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:20.872107 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:20.872199 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:20.872953 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:20.872956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:20.872959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:20.872970 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:20.872998 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:20.873010 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:20.873125 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:20.873233 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 12:34:20.873273 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:34:20.875958 21951 raft_consensus.cc:348] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e864300082b9413bbad78387420d876c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43911 } }
I20260619 12:34:20.876174 21951 raft_consensus.cc:374] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:20.876207 21951 raft_consensus.cc:729] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e864300082b9413bbad78387420d876c, State: Initialized, Role: FOLLOWER
I20260619 12:34:20.876613 21951 consensus_queue.cc:260] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e864300082b9413bbad78387420d876c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43911 } }
I20260619 12:34:20.876736 21951 raft_consensus.cc:388] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:20.876777 21951 raft_consensus.cc:482] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:20.876845 21951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:20.877748 21951 raft_consensus.cc:504] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e864300082b9413bbad78387420d876c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43911 } }
I20260619 12:34:20.878065 21951 leader_election.cc:302] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e864300082b9413bbad78387420d876c; no voters:
I20260619 12:34:20.878310 21951 leader_election.cc:288] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:20.878446 21961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:20.878732 21961 raft_consensus.cc:686] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c [term 1 LEADER]: Becoming Leader. State: Replica: e864300082b9413bbad78387420d876c, State: Running, Role: LEADER
I20260619 12:34:20.879108 21961 consensus_queue.cc:237] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e864300082b9413bbad78387420d876c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43911 } }
I20260619 12:34:20.879552 21951 sys_catalog.cc:564] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:20.880666 21962 sys_catalog.cc:455] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e864300082b9413bbad78387420d876c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e864300082b9413bbad78387420d876c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43911 } } }
I20260619 12:34:20.880803 21962 sys_catalog.cc:458] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:20.881234 21970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:20.881196 21963 sys_catalog.cc:455] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e864300082b9413bbad78387420d876c. Latest consensus state: current_term: 1 leader_uuid: "e864300082b9413bbad78387420d876c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e864300082b9413bbad78387420d876c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43911 } } }
I20260619 12:34:20.881336 21963 sys_catalog.cc:458] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:20.884250 21970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:20.889654 21970 catalog_manager.cc:1347] Generated new cluster ID: 71c4115913844128a2faed7b2335c648
I20260619 12:34:20.889721 21970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:20.939342 21970 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:20.939749 21970 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:20.952108 21970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e864300082b9413bbad78387420d876c: Generated new TSK 0
I20260619 12:34:20.952906 21970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:34:20.965062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 21883
2026-06-19T12:34:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:21 dist-test-slave-fg3h krb5kdc[21982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:21 dist-test-slave-fg3h krb5kdc[21982](info): set up 2 sockets
Jun 19 12:34:21 dist-test-slave-fg3h krb5kdc[21982](info): commencing operation
krb5kdc: starting...
W20260619 12:34:22.009404 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:22 dist-test-slave-fg3h krb5kdc[21982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:34:22.136494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36527
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:22.246517 21995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:22.246838 21995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:22.246922 21995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:22.250979 21995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:22.251082 21995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:22.251127 21995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:22.251188 21995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:22.251230 21995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:22.256430 21995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36527
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:22.257512 21995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:22.258541 21995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:22.267233 22000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:22.267218 22003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:22.267499 21995 server_base.cc:1034] running on GCE node
W20260619 12:34:22.267218 22001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:22.268285 21995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:22.269562 21995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:22.270746 21995 hybrid_clock.cc:648] HybridClock initialized: now 1781872462270729 us; error 81 us; skew 500 ppm
Jun 19 12:34:22 dist-test-slave-fg3h krb5kdc[21982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:22.273836 21995 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:34:22.274801 21995 webserver.cc:460] Webserver started at http://127.0.98.62:43161/ using document root <none> and password file <none>
I20260619 12:34:22.275404 21995 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:22.275499 21995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:22.275731 21995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:22.277900 21995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "abe066851c6046d68a887bc3d39c2270"
format_stamp: "Formatted at 2026-06-19 12:34:22 on dist-test-slave-fg3h"
server_key: "9007f21783b421eaa3f5e18bfee28c42"
server_key_iv: "a95084cb6617ac552d9f42bcc0f3bc67"
server_key_version: "encryptionkey@0"
I20260619 12:34:22.278509 21995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "abe066851c6046d68a887bc3d39c2270"
format_stamp: "Formatted at 2026-06-19 12:34:22 on dist-test-slave-fg3h"
server_key: "9007f21783b421eaa3f5e18bfee28c42"
server_key_iv: "a95084cb6617ac552d9f42bcc0f3bc67"
server_key_version: "encryptionkey@0"
I20260619 12:34:22.282400 21995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:34:22.285020 22010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:22.286341 21995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:34:22.286525 21995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "abe066851c6046d68a887bc3d39c2270"
format_stamp: "Formatted at 2026-06-19 12:34:22 on dist-test-slave-fg3h"
server_key: "9007f21783b421eaa3f5e18bfee28c42"
server_key_iv: "a95084cb6617ac552d9f42bcc0f3bc67"
server_key_version: "encryptionkey@0"
I20260619 12:34:22.286650 21995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:22.310127 21995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:22.313777 21995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:22.314006 21995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:22.321344 21995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36527
I20260619 12:34:22.321375 22062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36527 every 8 connection(s)
I20260619 12:34:22.322400 21995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:22.323305 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 21995
I20260619 12:34:22.323444 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:22.323758 392 external_mini_cluster.cc:1428] Setting key ba2dd83da99e0bc089dfcba1d4c8a668
I20260619 12:34:22.325878 22063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:22.331506 22063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270: Bootstrap starting.
Jun 19 12:34:22 dist-test-slave-fg3h krb5kdc[21982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:34:22.334142 22063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:22.334936 22063 log.cc:824] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:22.336838 22063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270: No bootstrap required, opened a new log
I20260619 12:34:22.339085 22066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:22.325289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53995 (local address 127.0.98.62:36527)
0619 12:34:22.325753 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:22.325769 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:22.325804 (+ 35us) server_negotiation.cc:407] Connection header received
0619 12:34:22.326569 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:22.326587 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:22.326958 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:22.327313 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:22.328172 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:22.329260 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:22.330145 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:22.330628 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:22.330813 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:22.334066 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:22.334073 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:22.334096 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:22.334129 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:22.336056 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:22.336599 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:22.336609 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:22.336613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:22.336723 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:22.337159 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:22.337162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:22.337169 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:22.337574 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:22.337794 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:22.338128 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 12:34:22.338351 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":161,"threads_started":1}
I20260619 12:34:22.340770 22063 raft_consensus.cc:348] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe066851c6046d68a887bc3d39c2270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } }
I20260619 12:34:22.341100 22063 raft_consensus.cc:374] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:22.341228 22063 raft_consensus.cc:729] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe066851c6046d68a887bc3d39c2270, State: Initialized, Role: FOLLOWER
I20260619 12:34:22.341666 22063 consensus_queue.cc:260] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe066851c6046d68a887bc3d39c2270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } }
I20260619 12:34:22.341835 22063 raft_consensus.cc:388] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:22.341899 22063 raft_consensus.cc:482] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:22.342005 22063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:22.342280 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:34:22.343297 22063 raft_consensus.cc:504] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe066851c6046d68a887bc3d39c2270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } }
I20260619 12:34:22.343685 22063 leader_election.cc:302] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abe066851c6046d68a887bc3d39c2270; no voters:
I20260619 12:34:22.344067 22063 leader_election.cc:288] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:22.344172 22072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:22.344427 22072 raft_consensus.cc:686] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270 [term 1 LEADER]: Becoming Leader. State: Replica: abe066851c6046d68a887bc3d39c2270, State: Running, Role: LEADER
I20260619 12:34:22.344803 22072 consensus_queue.cc:237] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe066851c6046d68a887bc3d39c2270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } }
I20260619 12:34:22.345516 22063 sys_catalog.cc:564] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:22.346577 22074 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abe066851c6046d68a887bc3d39c2270. Latest consensus state: current_term: 1 leader_uuid: "abe066851c6046d68a887bc3d39c2270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe066851c6046d68a887bc3d39c2270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } }
I20260619 12:34:22.346701 22074 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:22.347040 22073 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abe066851c6046d68a887bc3d39c2270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe066851c6046d68a887bc3d39c2270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36527 } } }
I20260619 12:34:22.347129 22073 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270 [sys.catalog]: This master's current role is: LEADER
W20260619 12:34:22.351011 22088 catalog_manager.cc:1558] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:34:22.351104 22088 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:34:22.351173 22082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:22.352173 22082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:22.354394 22066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:22.344898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41261 (local address 127.0.98.62:36527)
0619 12:34:22.345011 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:22.345016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:22.345031 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:34:22.345168 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:22.345171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:22.345216 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:22.345318 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:22.346161 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:22.347013 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:22.348271 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:22.348630 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:22.348744 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:22.350480 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:22.350483 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:22.350486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:22.350511 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:22.352275 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:22.352846 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:22.352849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:22.352851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:22.352942 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:22.353332 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:22.353336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:22.353337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:22.353561 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:22.353712 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:22.354109 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 12:34:22.354226 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 12:34:22.358327 22082 catalog_manager.cc:1347] Generated new cluster ID: a011a73bb92847e9a21ec7828eaf04be
I20260619 12:34:22.358392 22082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:22.368081 22082 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:22.369395 22082 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:22.384765 22082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abe066851c6046d68a887bc3d39c2270: Generated new TSK 0
I20260619 12:34:22.385667 22082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:34:22.396711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 21995
W20260619 12:34:22.402277 21980 connection.cc:535] client connection to 127.0.98.62:36527 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36527): Connection reset by peer (error 104)
2026-06-19T12:34:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:22 dist-test-slave-fg3h krb5kdc[22094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:22 dist-test-slave-fg3h krb5kdc[22094](info): set up 2 sockets
Jun 19 12:34:22 dist-test-slave-fg3h krb5kdc[22094](info): commencing operation
krb5kdc: starting...
W20260619 12:34:23.437389 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:23 dist-test-slave-fg3h krb5kdc[22094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:34:23.564471 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39455
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:23.676031 22107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:23.676291 22107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:23.676333 22107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:23.680318 22107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:23.680377 22107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:23.680397 22107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:23.680414 22107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:23.680460 22107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:23.685662 22107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39455
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:23.686725 22107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:23.687747 22107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:23.696302 22115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:23.696519 22107 server_base.cc:1034] running on GCE node
W20260619 12:34:23.696314 22112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:23.696388 22113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:23.697286 22107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:23.698406 22107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:23.699589 22107 hybrid_clock.cc:648] HybridClock initialized: now 1781872463699566 us; error 33 us; skew 500 ppm
Jun 19 12:34:23 dist-test-slave-fg3h krb5kdc[22094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:23.702570 22107 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:34:23.703567 22107 webserver.cc:460] Webserver started at http://127.0.98.62:33973/ using document root <none> and password file <none>
I20260619 12:34:23.704144 22107 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:23.704207 22107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:23.704424 22107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:23.706621 22107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "143d43863d424312b0771dc46ae443ca"
format_stamp: "Formatted at 2026-06-19 12:34:23 on dist-test-slave-fg3h"
server_key: "71439c42c9658c0e591f42f0f0e5764d"
server_key_iv: "1d1b09e8a4ac9b3009ba65c08ec4acbb"
server_key_version: "encryptionkey@0"
I20260619 12:34:23.707194 22107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "143d43863d424312b0771dc46ae443ca"
format_stamp: "Formatted at 2026-06-19 12:34:23 on dist-test-slave-fg3h"
server_key: "71439c42c9658c0e591f42f0f0e5764d"
server_key_iv: "1d1b09e8a4ac9b3009ba65c08ec4acbb"
server_key_version: "encryptionkey@0"
I20260619 12:34:23.711086 22107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:34:23.713544 22122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:23.714747 22107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:34:23.714897 22107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "143d43863d424312b0771dc46ae443ca"
format_stamp: "Formatted at 2026-06-19 12:34:23 on dist-test-slave-fg3h"
server_key: "71439c42c9658c0e591f42f0f0e5764d"
server_key_iv: "1d1b09e8a4ac9b3009ba65c08ec4acbb"
server_key_version: "encryptionkey@0"
I20260619 12:34:23.715022 22107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:23.731652 22107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:23.735095 22107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:23.735304 22107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:23.742594 22107 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39455
I20260619 12:34:23.742621 22174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39455 every 8 connection(s)
I20260619 12:34:23.743602 22107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:23.746302 22175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:23.751156 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 22107
I20260619 12:34:23.751291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:23.751607 392 external_mini_cluster.cc:1428] Setting key 5b69b668e34fa624733568dadacf5c67
I20260619 12:34:23.752087 22175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca: Bootstrap starting.
I20260619 12:34:23.755082 22175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:23.756034 22175 log.cc:824] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:23.758200 22175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca: No bootstrap required, opened a new log
I20260619 12:34:23.761463 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:23.753032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45665 (local address 127.0.98.62:39455)
0619 12:34:23.753531 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:23.753550 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:23.753595 (+ 45us) server_negotiation.cc:407] Connection header received
0619 12:34:23.754429 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:23.754453 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:23.754893 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:23.755253 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:23.755710 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:23.756992 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:23.758203 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:23.758732 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:23.758858 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:23.759754 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:23.759760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:23.759774 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:23.760158 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:23.760209 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:23.760231 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:23.760330 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:23.760534 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 12:34:23.760639 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":129,"threads_started":1}
I20260619 12:34:23.762430 22175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143d43863d424312b0771dc46ae443ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39455 } }
I20260619 12:34:23.762703 22175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:23.762790 22175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 143d43863d424312b0771dc46ae443ca, State: Initialized, Role: FOLLOWER
I20260619 12:34:23.763273 22175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143d43863d424312b0771dc46ae443ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39455 } }
I20260619 12:34:23.763437 22175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:23.763552 22175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:23.763681 22175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:23.764515 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:34:23.764808 22175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143d43863d424312b0771dc46ae443ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39455 } }
I20260619 12:34:23.765187 22175 leader_election.cc:302] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 143d43863d424312b0771dc46ae443ca; no voters:
I20260619 12:34:23.765486 22175 leader_election.cc:288] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:23.765584 22180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:23.765806 22180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca [term 1 LEADER]: Becoming Leader. State: Replica: 143d43863d424312b0771dc46ae443ca, State: Running, Role: LEADER
I20260619 12:34:23.766202 22180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143d43863d424312b0771dc46ae443ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39455 } }
I20260619 12:34:23.766726 22175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:23.767968 22181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "143d43863d424312b0771dc46ae443ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143d43863d424312b0771dc46ae443ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39455 } } }
I20260619 12:34:23.768111 22181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:23.768537 22182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 143d43863d424312b0771dc46ae443ca. Latest consensus state: current_term: 1 leader_uuid: "143d43863d424312b0771dc46ae443ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143d43863d424312b0771dc46ae443ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39455 } } }
I20260619 12:34:23.768671 22182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:23.771685 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:23.766827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38431 (local address 127.0.98.62:39455)
0619 12:34:23.766931 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:23.766935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:23.767396 (+ 461us) server_negotiation.cc:407] Connection header received
0619 12:34:23.767537 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:23.767540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:23.767584 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:23.767637 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:23.768132 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:23.768894 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:23.769991 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:23.770213 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:23.770322 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:23.771188 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:23.771195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:23.771197 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:23.771210 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:23.771235 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:23.771248 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:23.771339 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:23.771511 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 12:34:23.771546 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260619 12:34:23.771912 22200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:34:23.771986 22200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:34:23.772521 22196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:23.773331 22196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:23.779018 22196 catalog_manager.cc:1347] Generated new cluster ID: ec7bb6723732435aa94babb5dc4065a6
I20260619 12:34:23.779094 22196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:23.791754 22196 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:23.792944 22196 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:23.798326 22196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 143d43863d424312b0771dc46ae443ca: Generated new TSK 0
I20260619 12:34:23.799096 22196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:34:23.805187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 22107
W20260619 12:34:23.810333 22092 connection.cc:535] client connection to 127.0.98.62:39455 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39455): Connection reset by peer (error 104)
2026-06-19T12:34:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:23 dist-test-slave-fg3h krb5kdc[22206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:23 dist-test-slave-fg3h krb5kdc[22206](info): set up 2 sockets
Jun 19 12:34:23 dist-test-slave-fg3h krb5kdc[22206](info): commencing operation
krb5kdc: starting...
W20260619 12:34:24.869549 392 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:24 dist-test-slave-fg3h krb5kdc[22206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:34:24.996837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36917
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:25.114424 22219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:25.114779 22219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:25.114859 22219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:25.119007 22219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:25.119110 22219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:25.119150 22219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:25.119190 22219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:25.119231 22219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:25.124756 22219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36917
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:25.125880 22219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:25.126920 22219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:25.135365 22224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:25.135582 22225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:25.135473 22227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:25.135795 22219 server_base.cc:1034] running on GCE node
I20260619 12:34:25.136467 22219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:25.137629 22219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:25.138825 22219 hybrid_clock.cc:648] HybridClock initialized: now 1781872465138779 us; error 64 us; skew 500 ppm
Jun 19 12:34:25 dist-test-slave-fg3h krb5kdc[22206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:25.141667 22219 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:34:25.142541 22219 webserver.cc:460] Webserver started at http://127.0.98.62:40849/ using document root <none> and password file <none>
I20260619 12:34:25.143059 22219 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:25.143115 22219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:25.143338 22219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:25.145364 22219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "7c041848344d40e387c11159492cd36a"
format_stamp: "Formatted at 2026-06-19 12:34:25 on dist-test-slave-fg3h"
server_key: "276d7b8f979d0b23e9e4783e1ea9c1a9"
server_key_iv: "1b223ff16daf3f119f82ad0887289fe2"
server_key_version: "encryptionkey@0"
I20260619 12:34:25.145861 22219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "7c041848344d40e387c11159492cd36a"
format_stamp: "Formatted at 2026-06-19 12:34:25 on dist-test-slave-fg3h"
server_key: "276d7b8f979d0b23e9e4783e1ea9c1a9"
server_key_iv: "1b223ff16daf3f119f82ad0887289fe2"
server_key_version: "encryptionkey@0"
I20260619 12:34:25.150049 22219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 12:34:25.153620 22234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:25.155230 22219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 12:34:25.155366 22219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "7c041848344d40e387c11159492cd36a"
format_stamp: "Formatted at 2026-06-19 12:34:25 on dist-test-slave-fg3h"
server_key: "276d7b8f979d0b23e9e4783e1ea9c1a9"
server_key_iv: "1b223ff16daf3f119f82ad0887289fe2"
server_key_version: "encryptionkey@0"
I20260619 12:34:25.155503 22219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:25.162288 22219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:25.166678 22219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:25.166919 22219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:25.175295 22219 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36917
I20260619 12:34:25.175414 22286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36917 every 8 connection(s)
I20260619 12:34:25.176379 22219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:25.179246 22287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:25.184137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 22219
I20260619 12:34:25.184265 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:25.184574 392 external_mini_cluster.cc:1428] Setting key 0d4751a5bdb72109c3ce52143483eb83
I20260619 12:34:25.186332 22287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a: Bootstrap starting.
I20260619 12:34:25.189524 22287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:25.190531 22287 log.cc:824] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:25.191045 22290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:25.186053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34421 (local address 127.0.98.62:36917)
0619 12:34:25.186600 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:25.186620 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:25.186663 (+ 43us) server_negotiation.cc:407] Connection header received
0619 12:34:25.187737 (+ 1074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:25.187804 (+ 67us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:25.188246 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:25.188679 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:25.189476 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:25.189481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:25.189495 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:25.189507 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:25.189574 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:25.189602 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:25.189704 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:25.189915 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 12:34:25.190090 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":141,"threads_started":1}
I20260619 12:34:25.192811 22287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a: No bootstrap required, opened a new log
I20260619 12:34:25.194152 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:34:25.197996 22290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:25.196279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45103 (local address 127.0.98.62:36917)
0619 12:34:25.196402 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:25.196406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:25.196640 (+ 234us) server_negotiation.cc:407] Connection header received
0619 12:34:25.196758 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:25.196760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:25.196787 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:25.196877 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:25.197614 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:25.197617 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:25.197620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:25.197631 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:25.197658 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:25.197671 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:25.197747 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:25.197816 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 12:34:25.197853 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 12:34:25.197837 22287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c041848344d40e387c11159492cd36a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } }
I20260619 12:34:25.198148 22287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:25.198215 22287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c041848344d40e387c11159492cd36a, State: Initialized, Role: FOLLOWER
I20260619 12:34:25.198740 22287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c041848344d40e387c11159492cd36a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } }
I20260619 12:34:25.198921 22287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:25.199007 22287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:25.199128 22287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:25.200580 22287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c041848344d40e387c11159492cd36a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } }
I20260619 12:34:25.200995 22287 leader_election.cc:302] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c041848344d40e387c11159492cd36a; no voters:
I20260619 12:34:25.201377 22287 leader_election.cc:288] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:25.201438 22297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:25.201618 22297 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a [term 1 LEADER]: Becoming Leader. State: Replica: 7c041848344d40e387c11159492cd36a, State: Running, Role: LEADER
I20260619 12:34:25.201900 22297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c041848344d40e387c11159492cd36a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } }
I20260619 12:34:25.202618 22287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:25.203048 22297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c041848344d40e387c11159492cd36a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c041848344d40e387c11159492cd36a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } }
I20260619 12:34:25.203168 22297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:25.203374 22297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c041848344d40e387c11159492cd36a. Latest consensus state: current_term: 1 leader_uuid: "7c041848344d40e387c11159492cd36a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c041848344d40e387c11159492cd36a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } }
I20260619 12:34:25.203474 22297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:25.203626 22301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:25.207581 22301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:25.213402 22301 catalog_manager.cc:1347] Generated new cluster ID: e8fa8a8729f447f6a49bef554ffc77c6
I20260619 12:34:25.213457 22301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:25.230203 22301 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:25.230544 22301 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:25.236259 22301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c041848344d40e387c11159492cd36a: Generated new TSK 0
I20260619 12:34:25.236999 22301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:34:25.248791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 22219
2026-06-19T12:34:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:25 dist-test-slave-fg3h krb5kdc[22318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:25 dist-test-slave-fg3h krb5kdc[22318](info): set up 2 sockets
Jun 19 12:34:25 dist-test-slave-fg3h krb5kdc[22318](info): commencing operation
krb5kdc: starting...
W20260619 12:34:26.293366 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:26 dist-test-slave-fg3h krb5kdc[22318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:34:26.416656 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39227
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:26.526399 22331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:26.526717 22331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:26.526799 22331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:26.530833 22331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:26.530941 22331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:26.530988 22331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:26.531031 22331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:26.531067 22331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:26.536473 22331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:26.537611 22331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:26.538674 22331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:34:26.548058 22331 server_base.cc:1034] running on GCE node
W20260619 12:34:26.547863 22339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:26.548008 22336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:26.547863 22337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:26.548940 22331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:26.550117 22331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:26.551294 22331 hybrid_clock.cc:648] HybridClock initialized: now 1781872466551264 us; error 43 us; skew 500 ppm
Jun 19 12:34:26 dist-test-slave-fg3h krb5kdc[22318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:26.557241 22331 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:34:26.558320 22331 webserver.cc:460] Webserver started at http://127.0.98.62:42733/ using document root <none> and password file <none>
I20260619 12:34:26.558919 22331 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:26.558985 22331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:26.559166 22331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:26.561508 22331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "2cb30e1c135c45309efe0cb47e2b7e47"
format_stamp: "Formatted at 2026-06-19 12:34:26 on dist-test-slave-fg3h"
server_key: "c5a1731f5baab4e3ec89cc100e744edc"
server_key_iv: "e952c1386c5748d9b84566615abaf543"
server_key_version: "encryptionkey@0"
I20260619 12:34:26.562077 22331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "2cb30e1c135c45309efe0cb47e2b7e47"
format_stamp: "Formatted at 2026-06-19 12:34:26 on dist-test-slave-fg3h"
server_key: "c5a1731f5baab4e3ec89cc100e744edc"
server_key_iv: "e952c1386c5748d9b84566615abaf543"
server_key_version: "encryptionkey@0"
I20260619 12:34:26.566231 22331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:34:26.568812 22346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:26.570250 22331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:34:26.570430 22331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "2cb30e1c135c45309efe0cb47e2b7e47"
format_stamp: "Formatted at 2026-06-19 12:34:26 on dist-test-slave-fg3h"
server_key: "c5a1731f5baab4e3ec89cc100e744edc"
server_key_iv: "e952c1386c5748d9b84566615abaf543"
server_key_version: "encryptionkey@0"
I20260619 12:34:26.570564 22331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:26.587034 22331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:26.590489 22331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:26.590708 22331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:26.598223 22331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39227
I20260619 12:34:26.598258 22398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39227 every 8 connection(s)
I20260619 12:34:26.599207 22331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:26.601969 22399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:26.603466 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 22331
I20260619 12:34:26.603569 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:26.603891 392 external_mini_cluster.cc:1428] Setting key ef8b593571809ec9c6a3e63a245e64f6
I20260619 12:34:26.608013 22399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47: Bootstrap starting.
I20260619 12:34:26.610289 22399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:26.611001 22399 log.cc:824] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47: Log is configured to *not* fsync() on all Append() calls
Jun 19 12:34:26 dist-test-slave-fg3h krb5kdc[22318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:34:26.612879 22399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47: No bootstrap required, opened a new log
I20260619 12:34:26.616547 22399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb30e1c135c45309efe0cb47e2b7e47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39227 } }
I20260619 12:34:26.616791 22399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:26.616918 22399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb30e1c135c45309efe0cb47e2b7e47, State: Initialized, Role: FOLLOWER
I20260619 12:34:26.617424 22399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb30e1c135c45309efe0cb47e2b7e47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39227 } }
I20260619 12:34:26.617583 22399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:26.617678 22399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:26.617797 22399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:26.618269 22402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:26.605317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47077 (local address 127.0.98.62:39227)
0619 12:34:26.605798 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:26.605814 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:26.605851 (+ 37us) server_negotiation.cc:407] Connection header received
0619 12:34:26.606480 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:26.606495 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:26.606833 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:26.607171 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:26.608001 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:26.609027 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:26.609964 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:26.610486 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:26.610653 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:26.613174 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:26.613179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:26.613193 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:26.613220 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:26.615142 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:26.615627 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:26.615633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:26.615646 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:26.615758 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:26.616060 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:26.616063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:26.616065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:26.616459 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:26.616666 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:26.616970 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 12:34:26.617339 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":132,"threads_started":1}
I20260619 12:34:26.618880 22399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb30e1c135c45309efe0cb47e2b7e47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39227 } }
I20260619 12:34:26.619223 22399 leader_election.cc:302] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cb30e1c135c45309efe0cb47e2b7e47; no voters:
I20260619 12:34:26.619585 22399 leader_election.cc:288] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:26.619659 22404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:26.619856 22404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47 [term 1 LEADER]: Becoming Leader. State: Replica: 2cb30e1c135c45309efe0cb47e2b7e47, State: Running, Role: LEADER
I20260619 12:34:26.620500 22404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb30e1c135c45309efe0cb47e2b7e47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39227 } }
I20260619 12:34:26.620872 22399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:26.621567 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:34:26.622298 22406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cb30e1c135c45309efe0cb47e2b7e47. Latest consensus state: current_term: 1 leader_uuid: "2cb30e1c135c45309efe0cb47e2b7e47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb30e1c135c45309efe0cb47e2b7e47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39227 } } }
I20260619 12:34:26.622334 22405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cb30e1c135c45309efe0cb47e2b7e47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb30e1c135c45309efe0cb47e2b7e47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39227 } } }
I20260619 12:34:26.622421 22406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:26.622447 22405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47 [sys.catalog]: This master's current role is: LEADER
W20260619 12:34:26.625926 22420 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:34:26.626017 22420 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:34:26.626111 22415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:26.626976 22415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:26.633188 22402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:26.624218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34015 (local address 127.0.98.62:39227)
0619 12:34:26.624310 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:26.624315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:26.624440 (+ 125us) server_negotiation.cc:407] Connection header received
0619 12:34:26.624607 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:26.624610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:26.624653 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:26.624700 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:26.625620 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:26.626405 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:26.627532 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:26.627900 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:26.628012 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:26.629567 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:26.629571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:26.629574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:26.629599 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:26.631185 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:26.631714 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:26.631717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:26.631719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:26.631805 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:26.632105 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:26.632108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:26.632109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:26.632337 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:26.632473 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:26.632735 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 12:34:26.632865 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 12:34:26.633347 22415 catalog_manager.cc:1347] Generated new cluster ID: a1bfa6f976124f6ca7fe9f759ba9ec20
I20260619 12:34:26.633559 22415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:26.650264 22415 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:26.651587 22415 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:26.657344 22415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cb30e1c135c45309efe0cb47e2b7e47: Generated new TSK 0
I20260619 12:34:26.658140 22415 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:34:26.665386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 22331
2026-06-19T12:34:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:26 dist-test-slave-fg3h krb5kdc[22430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:26 dist-test-slave-fg3h krb5kdc[22430](info): set up 2 sockets
Jun 19 12:34:26 dist-test-slave-fg3h krb5kdc[22430](info): commencing operation
krb5kdc: starting...
W20260619 12:34:27.717533 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:27 dist-test-slave-fg3h krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:34:27.846051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38985
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:27.958979 22443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:27.959264 22443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:27.959306 22443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:27.963541 22443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:27.963608 22443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:27.963629 22443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:27.963645 22443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:27.963660 22443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:27.969645 22443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:27.970741 22443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:27.971873 22443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 12:34:27.980757 22443 server_base.cc:1034] running on GCE node
W20260619 12:34:27.980695 22451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:27.980732 22448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:27.980695 22449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:27.981640 22443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:27.982859 22443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:27.984046 22443 hybrid_clock.cc:648] HybridClock initialized: now 1781872467984028 us; error 37 us; skew 500 ppm
Jun 19 12:34:27 dist-test-slave-fg3h krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:27.986986 22443 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:34:27.987890 22443 webserver.cc:460] Webserver started at http://127.0.98.62:45347/ using document root <none> and password file <none>
I20260619 12:34:27.988422 22443 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:27.988492 22443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:27.988655 22443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:27.990885 22443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "57c832f615204c5f9dd81c53dc3d19c2"
format_stamp: "Formatted at 2026-06-19 12:34:27 on dist-test-slave-fg3h"
server_key: "d430d1b0b55f4522518080c0658cc7a7"
server_key_iv: "5310bdea57f022b89b4524772c2ca5a8"
server_key_version: "encryptionkey@0"
I20260619 12:34:27.991418 22443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "57c832f615204c5f9dd81c53dc3d19c2"
format_stamp: "Formatted at 2026-06-19 12:34:27 on dist-test-slave-fg3h"
server_key: "d430d1b0b55f4522518080c0658cc7a7"
server_key_iv: "5310bdea57f022b89b4524772c2ca5a8"
server_key_version: "encryptionkey@0"
I20260619 12:34:27.995074 22443 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 12:34:27.997563 22458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:27.998934 22443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:34:27.999042 22443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "57c832f615204c5f9dd81c53dc3d19c2"
format_stamp: "Formatted at 2026-06-19 12:34:27 on dist-test-slave-fg3h"
server_key: "d430d1b0b55f4522518080c0658cc7a7"
server_key_iv: "5310bdea57f022b89b4524772c2ca5a8"
server_key_version: "encryptionkey@0"
I20260619 12:34:27.999138 22443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:28.013849 22443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:28.017053 22443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:28.017318 22443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:28.024204 22443 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38985
I20260619 12:34:28.024291 22510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38985 every 8 connection(s)
I20260619 12:34:28.025301 22443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:28.028330 22511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:28.032894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 22443
I20260619 12:34:28.033041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:28.033352 392 external_mini_cluster.cc:1428] Setting key fe1afb9a9f756f087baaaaea4fa6ed8d
I20260619 12:34:28.034722 22511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2: Bootstrap starting.
I20260619 12:34:28.037058 22511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:28.037854 22511 log.cc:824] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:28.039772 22511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2: No bootstrap required, opened a new log
I20260619 12:34:28.043746 22514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:28.034979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47437 (local address 127.0.98.62:38985)
0619 12:34:28.035439 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:28.035454 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:28.035491 (+ 37us) server_negotiation.cc:407] Connection header received
0619 12:34:28.036192 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:28.036211 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:28.036564 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:28.036929 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:28.037387 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:28.038430 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:28.039365 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:28.039849 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:28.040017 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:28.041068 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:28.041075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:28.041118 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:28.041707 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:28.041779 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:28.041832 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:28.041943 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:28.042183 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 12:34:28.042633 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":152,"threads_started":1}
I20260619 12:34:28.044801 22511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c832f615204c5f9dd81c53dc3d19c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38985 } }
I20260619 12:34:28.045110 22511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:28.045168 22511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57c832f615204c5f9dd81c53dc3d19c2, State: Initialized, Role: FOLLOWER
I20260619 12:34:28.045825 22511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c832f615204c5f9dd81c53dc3d19c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38985 } }
I20260619 12:34:28.046020 22511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:28.046094 22511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:28.046190 22511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:28.047312 22511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c832f615204c5f9dd81c53dc3d19c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38985 } }
I20260619 12:34:28.047386 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:34:28.047701 22511 leader_election.cc:302] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57c832f615204c5f9dd81c53dc3d19c2; no voters:
I20260619 12:34:28.048064 22511 leader_election.cc:288] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:28.048135 22516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:28.048328 22516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2 [term 1 LEADER]: Becoming Leader. State: Replica: 57c832f615204c5f9dd81c53dc3d19c2, State: Running, Role: LEADER
I20260619 12:34:28.048641 22516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c832f615204c5f9dd81c53dc3d19c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38985 } }
I20260619 12:34:28.049434 22511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:28.049885 22516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57c832f615204c5f9dd81c53dc3d19c2. Latest consensus state: current_term: 1 leader_uuid: "57c832f615204c5f9dd81c53dc3d19c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c832f615204c5f9dd81c53dc3d19c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38985 } } }
I20260619 12:34:28.050004 22516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:28.050320 22517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57c832f615204c5f9dd81c53dc3d19c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c832f615204c5f9dd81c53dc3d19c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38985 } } }
I20260619 12:34:28.050438 22517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:28.050521 22525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:28.053925 22525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:28.056061 22514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:28.049641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46223 (local address 127.0.98.62:38985)
0619 12:34:28.049757 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:28.049761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:28.050896 (+ 1135us) server_negotiation.cc:407] Connection header received
0619 12:34:28.051097 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:28.051100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:28.051151 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:28.051204 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:28.051610 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:28.052862 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:28.054062 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:28.054421 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:28.054509 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:28.055577 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:28.055583 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:28.055586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:28.055601 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:28.055631 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:28.055644 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:28.055690 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:28.055857 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 12:34:28.055926 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 12:34:28.060772 22525 catalog_manager.cc:1347] Generated new cluster ID: f8d6de2bee334093a040fb276c190895
I20260619 12:34:28.060863 22525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:28.089358 22525 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:28.090443 22525 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:28.096088 22525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57c832f615204c5f9dd81c53dc3d19c2: Generated new TSK 0
I20260619 12:34:28.096938 22525 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:34:28.108588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 22443
2026-06-19T12:34:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:28 dist-test-slave-fg3h krb5kdc[22542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:28 dist-test-slave-fg3h krb5kdc[22542](info): set up 2 sockets
Jun 19 12:34:28 dist-test-slave-fg3h krb5kdc[22542](info): commencing operation
krb5kdc: starting...
W20260619 12:34:29.153424 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:29 dist-test-slave-fg3h krb5kdc[22542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:34:29.277119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42037
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:29.389686 22555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:29.389952 22555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:29.389994 22555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:29.394037 22555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:29.394095 22555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:29.394114 22555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:29.394131 22555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:29.394146 22555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:29.399286 22555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:29.400313 22555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:29.401371 22555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:29.410058 22560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:29.410063 22561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:29.410172 22555 server_base.cc:1034] running on GCE node
W20260619 12:34:29.410054 22563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:29.410915 22555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:29.412073 22555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:29.413237 22555 hybrid_clock.cc:648] HybridClock initialized: now 1781872469413193 us; error 35 us; skew 500 ppm
Jun 19 12:34:29 dist-test-slave-fg3h krb5kdc[22542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:29.416163 22555 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:34:29.417156 22555 webserver.cc:460] Webserver started at http://127.0.98.62:35887/ using document root <none> and password file <none>
I20260619 12:34:29.417744 22555 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:29.417811 22555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:29.418025 22555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:29.420104 22555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "39770055b6a2451db62dde6e1fb3cb62"
format_stamp: "Formatted at 2026-06-19 12:34:29 on dist-test-slave-fg3h"
server_key: "5d0e60829fb1bf3d90b94b820a148f7d"
server_key_iv: "3e1309ea83a441a77fd402ce90ce2ede"
server_key_version: "encryptionkey@0"
I20260619 12:34:29.420697 22555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "39770055b6a2451db62dde6e1fb3cb62"
format_stamp: "Formatted at 2026-06-19 12:34:29 on dist-test-slave-fg3h"
server_key: "5d0e60829fb1bf3d90b94b820a148f7d"
server_key_iv: "3e1309ea83a441a77fd402ce90ce2ede"
server_key_version: "encryptionkey@0"
I20260619 12:34:29.424568 22555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:34:29.427214 22570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:29.428694 22555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:34:29.428896 22555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "39770055b6a2451db62dde6e1fb3cb62"
format_stamp: "Formatted at 2026-06-19 12:34:29 on dist-test-slave-fg3h"
server_key: "5d0e60829fb1bf3d90b94b820a148f7d"
server_key_iv: "3e1309ea83a441a77fd402ce90ce2ede"
server_key_version: "encryptionkey@0"
I20260619 12:34:29.429021 22555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:29.433831 22555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:29.437058 22555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:29.437333 22555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:29.444546 22555 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42037
I20260619 12:34:29.444571 22622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42037 every 8 connection(s)
I20260619 12:34:29.445540 22555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:29.448338 22623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:29.453624 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 22555
I20260619 12:34:29.453815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:29.454008 22623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62: Bootstrap starting.
I20260619 12:34:29.454126 392 external_mini_cluster.cc:1428] Setting key 77244aa8b59b9517ba9361a8203ea557
I20260619 12:34:29.456487 22623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:29.457281 22623 log.cc:824] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:29.459396 22623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62: No bootstrap required, opened a new log
I20260619 12:34:29.459487 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:29.455495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57373 (local address 127.0.98.62:42037)
0619 12:34:29.455979 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:29.456005 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:29.456041 (+ 36us) server_negotiation.cc:407] Connection header received
0619 12:34:29.456756 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:29.456774 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:29.457061 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:29.457447 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:29.458236 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:29.458240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:29.458252 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:29.458260 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:29.458303 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:29.458320 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:29.458406 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:29.458550 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 12:34:29.458780 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":129,"threads_started":1}
I20260619 12:34:29.462482 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:34:29.463778 22623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39770055b6a2451db62dde6e1fb3cb62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42037 } }
I20260619 12:34:29.464089 22623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 12:34:29.464155 22623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39770055b6a2451db62dde6e1fb3cb62, State: Initialized, Role: FOLLOWER
I20260619 12:34:29.464661 22623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39770055b6a2451db62dde6e1fb3cb62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42037 } }
I20260619 12:34:29.464850 22623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 12:34:29.464936 22623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 12:34:29.465037 22623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:29.466116 22623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39770055b6a2451db62dde6e1fb3cb62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42037 } }
I20260619 12:34:29.466512 22623 leader_election.cc:302] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39770055b6a2451db62dde6e1fb3cb62; no voters:
I20260619 12:34:29.466709 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:29.464775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40787 (local address 127.0.98.62:42037)
0619 12:34:29.464895 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:29.464900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:29.465199 (+ 299us) server_negotiation.cc:407] Connection header received
0619 12:34:29.465337 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:29.465340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:29.465374 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:29.465476 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:29.466235 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:29.466239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:29.466245 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 12:34:29.466257 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:29.466286 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 12:34:29.466299 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:29.466379 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:29.466458 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 12:34:29.466497 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:34:29.466944 22623 leader_election.cc:288] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 12:34:29.467028 22633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:29.467278 22633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62 [term 1 LEADER]: Becoming Leader. State: Replica: 39770055b6a2451db62dde6e1fb3cb62, State: Running, Role: LEADER
I20260619 12:34:29.467617 22633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39770055b6a2451db62dde6e1fb3cb62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42037 } }
I20260619 12:34:29.468220 22623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:29.469394 22635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39770055b6a2451db62dde6e1fb3cb62. Latest consensus state: current_term: 1 leader_uuid: "39770055b6a2451db62dde6e1fb3cb62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39770055b6a2451db62dde6e1fb3cb62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42037 } } }
I20260619 12:34:29.469432 22634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39770055b6a2451db62dde6e1fb3cb62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39770055b6a2451db62dde6e1fb3cb62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42037 } } }
I20260619 12:34:29.469521 22635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:29.469527 22634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:29.469924 22643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:29.473134 22643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:29.478744 22643 catalog_manager.cc:1347] Generated new cluster ID: 977413d31e18441982b524c5e4a57e6c
I20260619 12:34:29.478847 22643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:29.496054 22643 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:29.496425 22643 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:29.507210 22643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39770055b6a2451db62dde6e1fb3cb62: Generated new TSK 0
I20260619 12:34:29.508178 22643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 12:34:29.523713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 22555
2026-06-19T12:34:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1414 ms)
[----------] 18 tests from AuthTokenIssuingTest (25784 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:29 dist-test-slave-fg3h krb5kdc[22654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:29 dist-test-slave-fg3h krb5kdc[22654](info): set up 2 sockets
Jun 19 12:34:29 dist-test-slave-fg3h krb5kdc[22654](info): commencing operation
krb5kdc: starting...
W20260619 12:34:30.569447 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:30 dist-test-slave-fg3h krb5kdc[22654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:34:30.696740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39849
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39849,127.0.98.61:39621,127.0.98.60:45343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:30.807461 22667 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 12:34:30.807782 22667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:30.807816 22667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:30.807910 22667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:30.812045 22667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:30.812106 22667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:30.812125 22667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:30.812141 22667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:30.812156 22667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:30.817610 22667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39849,127.0.98.61:39621,127.0.98.60:45343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:30.818750 22667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:30.819835 22667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:30.828666 22675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:30.828670 22672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:30.828660 22673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:30.828774 22667 server_base.cc:1034] running on GCE node
I20260619 12:34:30.829588 22667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:30.830778 22667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:30.831969 22667 hybrid_clock.cc:648] HybridClock initialized: now 1781872470831957 us; error 27 us; skew 500 ppm
Jun 19 12:34:30 dist-test-slave-fg3h krb5kdc[22654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:30.834954 22667 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:34:30.835930 22667 webserver.cc:460] Webserver started at http://127.0.98.62:41057/ using document root <none> and password file <none>
I20260619 12:34:30.836514 22667 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:30.836606 22667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:30.836844 22667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:30.839018 22667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "c34cf1fe5f1543d3b79fa044af87d312"
format_stamp: "Formatted at 2026-06-19 12:34:30 on dist-test-slave-fg3h"
server_key: "9dec6b815c80767689f4c7b193518465"
server_key_iv: "07c5c91d586b5732d2c146a1d2db548a"
server_key_version: "encryptionkey@0"
I20260619 12:34:30.839600 22667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "c34cf1fe5f1543d3b79fa044af87d312"
format_stamp: "Formatted at 2026-06-19 12:34:30 on dist-test-slave-fg3h"
server_key: "9dec6b815c80767689f4c7b193518465"
server_key_iv: "07c5c91d586b5732d2c146a1d2db548a"
server_key_version: "encryptionkey@0"
I20260619 12:34:30.844000 22667 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:34:30.846839 22682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:30.848337 22667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 12:34:30.848536 22667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "c34cf1fe5f1543d3b79fa044af87d312"
format_stamp: "Formatted at 2026-06-19 12:34:30 on dist-test-slave-fg3h"
server_key: "9dec6b815c80767689f4c7b193518465"
server_key_iv: "07c5c91d586b5732d2c146a1d2db548a"
server_key_version: "encryptionkey@0"
I20260619 12:34:30.848680 22667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:30.863480 22667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:30.867151 22667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:30.867475 22667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:30.875257 22667 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39849
I20260619 12:34:30.875553 22734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39849 every 8 connection(s)
I20260619 12:34:30.876300 22667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:30.879231 22735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:30.883407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 22667
I20260619 12:34:30.883566 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:30.883311 22735 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:34:30.884011 392 external_mini_cluster.cc:1428] Setting key b7c641ab76aa5c5ca3deed9bb97bae4f
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 12:34:30 dist-test-slave-fg3h krb5kdc[22654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:34:30.902086 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:30.886071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58440 (local address 127.0.98.62:39849)
0619 12:34:30.886630 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:30.886645 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:30.886702 (+ 57us) server_negotiation.cc:407] Connection header received
0619 12:34:30.887894 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:30.887899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:30.888090 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:30.888296 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:30.889195 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:30.890067 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:30.891048 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:30.891413 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:30.891578 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:30.896553 (+ 4975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:30.896558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:30.896568 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:30.896590 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:30.898781 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:30.899290 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:30.899298 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:30.899304 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:30.899407 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:30.899744 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:30.899747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:30.899749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:30.900143 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:30.900295 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:30.900661 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 12:34:30.900965 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":428,"spinlock_wait_cycles":22272,"thread_start_us":170,"threads_started":1}
I20260619 12:34:30.902082 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:30.886071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39849 (local address 127.0.0.1:58440)
0619 12:34:30.886555 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:30.886608 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:30.887499 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:30.888298 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:30.888303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:30.888750 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:30.889000 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:30.889013 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:30.890202 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:30.890205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:30.890929 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:30.890937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:30.891564 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:30.891568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:30.891673 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:30.892280 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:30.892296 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:30.896389 (+ 4093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:30.898903 (+ 2514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:30.898913 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:30.898925 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:30.899173 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:30.899507 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:30.899512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:30.899514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:30.899641 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:30.900275 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:30.900281 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:30.900424 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 12:34:30.900604 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 12:34:30.901012 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":112,"thread_start_us":147,"threads_started":1}
I20260619 12:34:30.905030 22735 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:34:30.905932 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:30.905580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39621 (local address 127.0.0.1:59310)
0619 12:34:30.905761 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:30.905862 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 12:34:30.906147 22686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39621: connect: Connection refused (error 111)
I20260619 12:34:30.907024 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:30.906818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39621 (local address 127.0.0.1:59318)
0619 12:34:30.906921 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:30.906971 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 12:34:30.907191 22735 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39621: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39621: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 12:34:30.912104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39621
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39849,127.0.98.61:39621,127.0.98.60:45343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:34:30.956455 22735 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } attempt: 1
I20260619 12:34:30.957463 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:30.957066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39621 (local address 127.0.0.1:59330)
0619 12:34:30.957289 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:30.957355 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260619 12:34:30.958395 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:30.958150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39621 (local address 127.0.0.1:59342)
0619 12:34:30.958261 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:30.958311 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 12:34:30.958614 22735 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39621: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39621: connect: Connection refused (error 111)
I20260619 12:34:31.026803 22735 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } attempt: 2
W20260619 12:34:31.027314 22742 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 12:34:31.027761 22742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:31.027838 22742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260619 12:34:31.027884 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.027418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39621 (local address 127.0.0.1:59346)
0619 12:34:31.027599 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.027661 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260619 12:34:31.027912 22742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 12:34:31.028694 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.028446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39621 (local address 127.0.0.1:59350)
0619 12:34:31.028574 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.028635 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 12:34:31.028833 22735 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39621: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39621: connect: Connection refused (error 111)
W20260619 12:34:31.032406 22742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:31.032495 22742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:31.032543 22742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:31.032578 22742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:31.032614 22742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:31.038151 22742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39849,127.0.98.61:39621,127.0.98.60:45343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:31.039288 22742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:31.040357 22742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:31.049115 22748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:31.049389 22750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:31.049355 22742 server_base.cc:1034] running on GCE node
W20260619 12:34:31.049314 22747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:31.050089 22742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:31.051302 22742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:31.052532 22742 hybrid_clock.cc:648] HybridClock initialized: now 1781872471052496 us; error 48 us; skew 500 ppm
Jun 19 12:34:31 dist-test-slave-fg3h krb5kdc[22654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:31.056505 22742 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 12:34:31.057487 22742 webserver.cc:460] Webserver started at http://127.0.98.61:35717/ using document root <none> and password file <none>
I20260619 12:34:31.058022 22742 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:31.058079 22742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:31.058241 22742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:31.060322 22742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-1/data/instance:
uuid: "712eea69af18486396ca2ab357499ae7"
format_stamp: "Formatted at 2026-06-19 12:34:31 on dist-test-slave-fg3h"
server_key: "8b83851f02e677f6342d4d92827249b8"
server_key_iv: "7dd2b448dad73800eec713f384183e1f"
server_key_version: "encryptionkey@0"
I20260619 12:34:31.060850 22742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-1/wal/instance:
uuid: "712eea69af18486396ca2ab357499ae7"
format_stamp: "Formatted at 2026-06-19 12:34:31 on dist-test-slave-fg3h"
server_key: "8b83851f02e677f6342d4d92827249b8"
server_key_iv: "7dd2b448dad73800eec713f384183e1f"
server_key_version: "encryptionkey@0"
I20260619 12:34:31.064587 22742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 12:34:31.067201 22757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:31.068828 22742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 12:34:31.068934 22742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-1/wal
uuid: "712eea69af18486396ca2ab357499ae7"
format_stamp: "Formatted at 2026-06-19 12:34:31 on dist-test-slave-fg3h"
server_key: "8b83851f02e677f6342d4d92827249b8"
server_key_iv: "7dd2b448dad73800eec713f384183e1f"
server_key_version: "encryptionkey@0"
I20260619 12:34:31.069026 22742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:31.089287 22742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:31.092404 22742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:31.092566 22742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:31.099932 22742 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39621
I20260619 12:34:31.100224 22809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39621 every 8 connection(s)
I20260619 12:34:31.100968 22742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-1/data/info.pb
I20260619 12:34:31.103915 22810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:31.107779 22810 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:34:31.108825 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 22742
I20260619 12:34:31.108966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-1/wal/instance
I20260619 12:34:31.109302 392 external_mini_cluster.cc:1428] Setting key a1a9af3528cc5ddc1e0767b8a8586392
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260619 12:34:31.120059 22735 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } attempt: 3
Jun 19 12:34:31 dist-test-slave-fg3h krb5kdc[22654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 12:34:31 dist-test-slave-fg3h krb5kdc[22654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 12:34:31.127346 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.110008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58452 (local address 127.0.98.62:39849)
0619 12:34:31.110182 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.110188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.111097 (+ 909us) server_negotiation.cc:407] Connection header received
0619 12:34:31.112083 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.112090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.112161 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.112273 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.113585 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.114528 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.115747 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.116171 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.116303 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.121502 (+ 5199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:31.121505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:31.121507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:31.121532 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:31.124419 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.124985 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.124990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.124992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.125073 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.125486 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.125491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.125493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.125739 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:31.126782 (+ 1043us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:31.126892 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 12:34:31.127047 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 12:34:31.127830 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.110423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39849 (local address 127.0.0.1:58452)
0619 12:34:31.110915 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.110969 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:31.111867 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:31.112449 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:31.112455 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:31.112822 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:31.113409 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:31.113423 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.114696 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.114699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:31.115604 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:31.115613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.116367 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.116371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:31.116539 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.117326 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:31.117345 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:31.121313 (+ 3968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:31.124549 (+ 3236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:31.124557 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:31.124575 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:31.124875 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:31.125221 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:31.125224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:31.125226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:31.125387 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:31.125874 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:31.125882 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:31.126255 (+ 373us) client_negotiation.cc:769] Sending connection context
0619 12:34:31.126414 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 12:34:31.126700 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":140,"threads_started":1}
I20260619 12:34:31.129753 22810 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:34:31.132747 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.120703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39621 (local address 127.0.0.1:59356)
0619 12:34:31.120940 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.120958 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:31.121050 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:31.121952 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:31.121956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:31.122178 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:31.122355 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:31.122364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.123579 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.123585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:31.124187 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:31.124196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.125256 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.125264 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:31.125425 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.125988 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:31.126011 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:31.127958 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:31.131119 (+ 3161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:31.131122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:31.131125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:31.131387 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:31.131693 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:31.131696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:31.131698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:31.131788 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:31.132231 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:31.132234 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:31.132332 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 12:34:31.132453 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 12:34:31.132592 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260619 12:34:31.132973 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.120888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59356 (local address 127.0.98.61:39621)
0619 12:34:31.121433 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.121448 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.121477 (+ 29us) server_negotiation.cc:407] Connection header received
0619 12:34:31.121550 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.121559 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.121797 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.121953 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.122506 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.123416 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.124328 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.125046 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.125695 (+ 649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.128543 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:31.128551 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:31.128566 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:31.128600 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:31.130985 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.131498 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.131504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.131506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.131591 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.131883 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.131886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.131888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.132122 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:31.132265 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:31.132550 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 12:34:31.132750 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":151,"threads_started":1}
I20260619 12:34:31.135144 22735 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:34:31 dist-test-slave-fg3h krb5kdc[22654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 12:34:31.135953 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.135620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45343 (local address 127.0.0.1:59108)
0619 12:34:31.135808 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.135878 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 12:34:31.136811 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.136501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45343 (local address 127.0.0.1:59112)
0619 12:34:31.136690 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.136744 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260619 12:34:31.136977 22735 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45343: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
I20260619 12:34:31.139613 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.130299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39621 (local address 127.0.0.1:59362)
0619 12:34:31.130424 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.130438 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:31.130517 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:31.130915 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:31.130918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:31.131096 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:31.131203 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:31.131213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.132322 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.132328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:31.133177 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:31.133188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.134178 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.134181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:31.134267 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.134735 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:31.134777 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:31.136269 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:31.138145 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:31.138149 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:31.138151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:31.138369 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:31.138644 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:31.138649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:31.138651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:31.138731 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:31.139100 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:31.139103 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:31.139187 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 12:34:31.139291 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 12:34:31.139436 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 12:34:31.139663 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.130424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59362 (local address 127.0.98.61:39621)
0619 12:34:31.130729 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.130735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.130750 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:34:31.130776 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.130778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.130811 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.130906 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.131330 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.132187 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.133294 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.134052 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.134158 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.136433 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:31.136437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:31.136439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:31.136461 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:31.138023 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.138468 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.138475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.138476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.138556 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.138817 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.138823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.138825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.139002 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:31.139100 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:31.139334 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 12:34:31.139495 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":255,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 12:34:31.140494 22810 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:34:31.141139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45343
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39849,127.0.98.61:39621,127.0.98.60:45343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:34:31.141374 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.141003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45343 (local address 127.0.0.1:59114)
0619 12:34:31.141168 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.141295 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 12:34:31.141573 22760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
I20260619 12:34:31.142379 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.142191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45343 (local address 127.0.0.1:59126)
0619 12:34:31.142289 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.142331 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 12:34:31.142565 22810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45343: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
I20260619 12:34:31.168191 22735 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } attempt: 1
I20260619 12:34:31.169173 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.168805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45343 (local address 127.0.0.1:59136)
0619 12:34:31.168960 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.169032 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 12:34:31.170085 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.169808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45343 (local address 127.0.0.1:59142)
0619 12:34:31.169967 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.170017 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260619 12:34:31.170275 22735 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45343: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
I20260619 12:34:31.191872 22810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } attempt: 1
I20260619 12:34:31.192852 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.192530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45343 (local address 127.0.0.1:59144)
0619 12:34:31.192699 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.192768 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 12:34:31.193708 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.193471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45343 (local address 127.0.0.1:59158)
0619 12:34:31.193604 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.193652 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 12:34:31.193918 22810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45343: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
I20260619 12:34:31.245462 22735 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } attempt: 2
I20260619 12:34:31.246305 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.245957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45343 (local address 127.0.0.1:59162)
0619 12:34:31.246141 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.246211 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 12:34:31.247184 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.246942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45343 (local address 127.0.0.1:59166)
0619 12:34:31.247059 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.247116 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 12:34:31.247357 22735 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45343: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
W20260619 12:34:31.256382 22818 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 12:34:31.256769 22818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:31.256836 22818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:31.256916 22818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:31.261651 22818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:31.261729 22818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:31.261751 22818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:31.261793 22818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:31.261831 22818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:31.262079 22810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } attempt: 2
I20260619 12:34:31.263021 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.262675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45343 (local address 127.0.0.1:59178)
0619 12:34:31.262845 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.262943 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 12:34:31.263803 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.263584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45343 (local address 127.0.0.1:59186)
0619 12:34:31.263699 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.263751 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 12:34:31.263942 22810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45343: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
I20260619 12:34:31.268860 22818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39849,127.0.98.61:39621,127.0.98.60:45343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:31.270085 22818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:31.271183 22818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:31.280503 22824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:31.280485 22823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:31.280524 22826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:31.280918 22818 server_base.cc:1034] running on GCE node
I20260619 12:34:31.281599 22818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:31.282811 22818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:31.284025 22818 hybrid_clock.cc:648] HybridClock initialized: now 1781872471283994 us; error 50 us; skew 500 ppm
Jun 19 12:34:31 dist-test-slave-fg3h krb5kdc[22654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:31.286993 22818 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 12:34:31.288021 22818 webserver.cc:460] Webserver started at http://127.0.98.60:33853/ using document root <none> and password file <none>
I20260619 12:34:31.288618 22818 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:31.288710 22818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:31.288959 22818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:31.291066 22818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-2/data/instance:
uuid: "56d55861702a49fdba9618ee9aa33ecf"
format_stamp: "Formatted at 2026-06-19 12:34:31 on dist-test-slave-fg3h"
server_key: "175fc268cfc1d01959154f39ac46aab5"
server_key_iv: "ae520a2f99bd898bb823b2c3e06984ec"
server_key_version: "encryptionkey@0"
I20260619 12:34:31.291667 22818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-2/wal/instance:
uuid: "56d55861702a49fdba9618ee9aa33ecf"
format_stamp: "Formatted at 2026-06-19 12:34:31 on dist-test-slave-fg3h"
server_key: "175fc268cfc1d01959154f39ac46aab5"
server_key_iv: "ae520a2f99bd898bb823b2c3e06984ec"
server_key_version: "encryptionkey@0"
I20260619 12:34:31.295485 22818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:34:31.298125 22833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:31.299396 22818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 12:34:31.299541 22818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-2/wal
uuid: "56d55861702a49fdba9618ee9aa33ecf"
format_stamp: "Formatted at 2026-06-19 12:34:31 on dist-test-slave-fg3h"
server_key: "175fc268cfc1d01959154f39ac46aab5"
server_key_iv: "ae520a2f99bd898bb823b2c3e06984ec"
server_key_version: "encryptionkey@0"
I20260619 12:34:31.299661 22818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:31.337630 22818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:31.340971 22818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:31.341219 22818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:31.346580 22735 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } attempt: 3
I20260619 12:34:31.347396 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.347077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45343 (local address 127.0.0.1:59190)
0619 12:34:31.347208 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.347291 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 12:34:31.348212 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.347993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45343 (local address 127.0.0.1:59200)
0619 12:34:31.348109 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.348155 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 12:34:31.348423 22735 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45343: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45343: connect: Connection refused (error 111)
I20260619 12:34:31.349242 22818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45343
I20260619 12:34:31.349532 22885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45343 every 8 connection(s)
I20260619 12:34:31.350230 22818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-2/data/info.pb
I20260619 12:34:31.353015 22886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:31.355093 22810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } attempt: 3
I20260619 12:34:31.357321 22886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:34:31.358587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 22818
I20260619 12:34:31.358724 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781872188319340-392-0/minicluster-data/master-2/wal/instance
I20260619 12:34:31.359028 392 external_mini_cluster.cc:1428] Setting key 3d75e842e5ebfa33733f6513866c809f
Jun 19 12:34:31 dist-test-slave-fg3h krb5kdc[22654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 12:34:31 dist-test-slave-fg3h krb5kdc[22654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 12:34:31 dist-test-slave-fg3h krb5kdc[22654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:34:31.368948 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.355596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45343 (local address 127.0.0.1:59202)
0619 12:34:31.355749 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.355766 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:31.355872 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:31.358333 (+ 2461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:31.358337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:31.358545 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:31.358669 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:31.358677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.360464 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.360468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:31.361014 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:31.361023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.361807 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.361812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:31.361907 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.362526 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:31.362542 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:31.364113 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:31.366779 (+ 2666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:31.366783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:31.366785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:31.367078 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:31.367477 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:31.367481 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:31.367483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:31.367587 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:31.368353 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:31.368356 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:31.368468 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 12:34:31.368585 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 12:34:31.368744 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 12:34:31.371462 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.356359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59202 (local address 127.0.98.60:45343)
0619 12:34:31.356800 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.356815 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.356881 (+ 66us) server_negotiation.cc:407] Connection header received
0619 12:34:31.357613 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.357631 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.357978 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.358345 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.358815 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.360342 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.361192 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.361673 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.361966 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.364276 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:31.364283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:31.364294 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:31.364319 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:31.366646 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.367193 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.367201 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.367211 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.367353 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.367726 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.367732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.367735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.368242 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:31.368382 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:31.370068 (+ 1686us) server_negotiation.cc:299] Negotiation successful
0619 12:34:31.370398 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":306,"thread_start_us":141,"threads_started":1}
I20260619 12:34:31.371840 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.359532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39849 (local address 127.0.0.1:58464)
0619 12:34:31.359936 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.359982 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:31.360175 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:31.361232 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:31.361241 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:31.361719 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:31.362002 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:31.362011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.363409 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.363414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:31.364308 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:31.364316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.365547 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.365550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:31.365638 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.366128 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:31.366148 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:31.367926 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:31.370019 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:31.370028 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:31.370035 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:31.370352 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:31.370685 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:31.370687 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:31.370688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:31.370758 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:31.371177 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:31.371187 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:31.371300 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 12:34:31.371441 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 12:34:31.371631 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":84,"spinlock_wait_cycles":12160,"thread_start_us":152,"threads_started":1}
I20260619 12:34:31.371920 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.359574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58464 (local address 127.0.98.62:39849)
0619 12:34:31.359685 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.359690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.360056 (+ 366us) server_negotiation.cc:407] Connection header received
0619 12:34:31.360332 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.360336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.360398 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.360488 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.362546 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.363292 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.364963 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.365408 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.365533 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.368087 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:31.368093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:31.368097 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:31.368133 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:31.369917 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.370482 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.370486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.370488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.370574 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.370873 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.370876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.370878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.371062 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:31.371169 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:31.371631 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 12:34:31.371763 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:34:31.371879 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.360321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59212 (local address 127.0.98.60:45343)
0619 12:34:31.360610 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.360618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.360628 (+ 10us) server_negotiation.cc:407] Connection header received
0619 12:34:31.360654 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.360656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.360685 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.360788 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.361670 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.362604 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.363624 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.363964 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.364066 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.367126 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:31.367131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:31.367136 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:31.367167 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:31.369710 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.370190 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.370193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.370195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.370276 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.370579 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.370582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.370583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.370776 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:31.370937 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:31.371562 (+ 625us) server_negotiation.cc:299] Negotiation successful
0619 12:34:31.371687 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":220,"thread_start_us":85,"threads_started":1}
I20260619 12:34:31.373801 22886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:34:31.376030 22810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7: Bootstrap starting.
I20260619 12:34:31.378671 22810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:31.379595 22810 log.cc:824] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:31.382517 22810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7: No bootstrap required, opened a new log
Jun 19 12:34:31 dist-test-slave-fg3h krb5kdc[22654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 12:34:31 dist-test-slave-fg3h krb5kdc[22654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:34:31.386510 22810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } peers { permanent_uuid: "712eea69af18486396ca2ab357499ae7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } } peers { permanent_uuid: "56d55861702a49fdba9618ee9aa33ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } }
I20260619 12:34:31.386911 22810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 712eea69af18486396ca2ab357499ae7, State: Initialized, Role: FOLLOWER
I20260619 12:34:31.387320 22810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } peers { permanent_uuid: "712eea69af18486396ca2ab357499ae7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } } peers { permanent_uuid: "56d55861702a49fdba9618ee9aa33ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } }
I20260619 12:34:31.387853 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.377253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59376 (local address 127.0.98.61:39621)
0619 12:34:31.377403 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.377408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.377425 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:34:31.377471 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.377475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.377525 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.377622 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.378873 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.379624 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.380525 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.380833 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.380952 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.383312 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:31.383316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:31.383319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:31.383345 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:31.385181 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.386417 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.386421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.386422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.386510 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.386789 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.386793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.386796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.387096 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:31.387203 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:31.387518 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 12:34:31.387644 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:34:31.388062 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.374383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58470 (local address 127.0.98.62:39849)
0619 12:34:31.377425 (+ 3042us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.377429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.377445 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:34:31.377482 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.377485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.377525 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.377622 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.378383 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.379015 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.379940 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.380316 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.380836 (+ 520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.384107 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:31.384111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:31.384113 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:31.384147 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:31.385864 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.386640 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.386643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.386645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.386761 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.387096 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.387100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.387102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.387329 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:31.387569 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:31.387730 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 12:34:31.387869 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2954}
I20260619 12:34:31.388476 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.374348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39621 (local address 127.0.0.1:59376)
0619 12:34:31.374557 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.374569 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:31.374641 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:31.378414 (+ 3773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:31.378417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:31.378609 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:31.378730 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:31.378740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.379776 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.379779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:31.380394 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:31.380402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.380951 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.380955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:31.381047 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.381557 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:31.381571 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:31.383154 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:31.386028 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:31.386032 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:31.386035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:31.386301 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:31.386599 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:31.386601 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:31.386602 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:31.386680 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:31.387204 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:31.387207 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:31.387315 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 12:34:31.388164 (+ 849us) client_negotiation.cc:240] Negotiation successful
0619 12:34:31.388322 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260619 12:34:31.389042 22810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:31.389396 22886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:34:31 dist-test-slave-fg3h krb5kdc[22654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
W20260619 12:34:31.395613 22904 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:34:31.395749 22904 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 19 12:34:31 dist-test-slave-fg3h krb5kdc[22654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 12:34:31.392694 22742 master_runner.cc:186] Error getting master registration for 127.0.98.62:39849: OK, instance_id { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" instance_seqno: 1781872470873040 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 12:34:31.396478 22742 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 12:34:31.396692 22893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } peers { permanent_uuid: "712eea69af18486396ca2ab357499ae7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } } peers { permanent_uuid: "56d55861702a49fdba9618ee9aa33ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } } }
I20260619 12:34:31.396871 22893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:34:31.399647 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.390062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59216 (local address 127.0.98.60:45343)
0619 12:34:31.390172 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.390177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.390346 (+ 169us) server_negotiation.cc:407] Connection header received
0619 12:34:31.390445 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.390452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.390511 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.390600 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.391062 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.391799 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.392746 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.393075 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.393192 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.395593 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:31.395597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:31.395599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:31.395620 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:31.397433 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.398191 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.398195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.398197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.398307 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.398640 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.398644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.398646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.398950 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:31.399063 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:31.399371 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 12:34:31.399463 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 12:34:31.400099 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.389917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45343 (local address 127.0.0.1:59216)
0619 12:34:31.390238 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.390261 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:31.390354 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:31.390629 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:31.390633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:31.390822 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:31.390938 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:31.390946 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.391918 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.391921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:31.392633 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:31.392642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.393333 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.393336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:31.393410 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.393924 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:31.393941 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:31.395461 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:31.397542 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:31.397548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:31.397552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:31.397807 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:31.398421 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:31.398425 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:31.398430 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:31.398530 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:31.399070 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:31.399076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:31.399178 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 12:34:31.399831 (+ 653us) client_negotiation.cc:240] Negotiation successful
0619 12:34:31.399958 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274}
I20260619 12:34:31.400844 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.389760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59380 (local address 127.0.98.61:39621)
0619 12:34:31.389887 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.389894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.389910 (+ 16us) server_negotiation.cc:407] Connection header received
0619 12:34:31.389946 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.389949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.389993 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.390941 (+ 948us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.391044 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.391851 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.392950 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.393319 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.393787 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.396543 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:31.396548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:31.396550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:31.396576 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:31.398710 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.399143 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.399146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.399148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.399225 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.399549 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.399552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.399554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.399768 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:31.399969 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:31.400571 (+ 602us) server_negotiation.cc:299] Negotiation successful
0619 12:34:31.400683 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:34:31.403620 22886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf: Bootstrap starting.
I20260619 12:34:31.404304 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:34:31.406654 22886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:31.405874 22709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c34cf1fe5f1543d3b79fa044af87d312"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58470
I20260619 12:34:31.407586 22886 log.cc:824] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:31.410130 22886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf: No bootstrap required, opened a new log
I20260619 12:34:31.413344 22886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } peers { permanent_uuid: "712eea69af18486396ca2ab357499ae7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } } peers { permanent_uuid: "56d55861702a49fdba9618ee9aa33ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } }
I20260619 12:34:31.413591 22886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d55861702a49fdba9618ee9aa33ecf, State: Initialized, Role: FOLLOWER
I20260619 12:34:31.413957 22886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } peers { permanent_uuid: "712eea69af18486396ca2ab357499ae7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } } peers { permanent_uuid: "56d55861702a49fdba9618ee9aa33ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } }
I20260619 12:34:31.414465 22907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } peers { permanent_uuid: "712eea69af18486396ca2ab357499ae7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } } peers { permanent_uuid: "56d55861702a49fdba9618ee9aa33ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } } }
I20260619 12:34:31.414603 22907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:34:31.415266 22886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf [sys.catalog]: configured and running, proceeding with master startup.
W20260619 12:34:31.421036 22918 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:34:31.421211 22918 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:34:31.419024 22818 master_runner.cc:186] Error getting master registration for 127.0.98.62:39849: OK, instance_id { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" instance_seqno: 1781872470873040 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 12:34:31.422359 22818 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 12:34:31.512609 22735 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } attempt: 4
Jun 19 12:34:31 dist-test-slave-fg3h krb5kdc[22654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 12:34:31.522213 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.513278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45343 (local address 127.0.0.1:59220)
0619 12:34:31.513503 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:31.513521 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:31.513635 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:31.513948 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:31.513951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:31.514181 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:31.514304 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:31.514319 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.515336 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.515339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:31.515897 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:31.515906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.516507 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.516511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:31.516599 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.517214 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:31.517230 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:31.518765 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:31.520586 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:31.520591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:31.520595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:31.520854 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:31.521171 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:31.521176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:31.521181 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:31.521274 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:31.521668 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:31.521671 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:31.521779 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 12:34:31.521882 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 12:34:31.522028 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260619 12:34:31.522213 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.513345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59220 (local address 127.0.98.60:45343)
0619 12:34:31.513480 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.513485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.513589 (+ 104us) server_negotiation.cc:407] Connection header received
0619 12:34:31.513759 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.513763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.513816 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.513918 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.514451 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.515220 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.516035 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.516404 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.516499 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.518923 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:31.518927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:31.518929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:31.518962 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:31.520450 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.520945 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.520948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.520950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.521032 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.521374 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.521377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.521379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.521566 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:31.521678 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:31.521944 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 12:34:31.522076 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 12:34:31.525970 22735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312: Bootstrap starting.
I20260619 12:34:31.528822 22735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:31.529736 22735 log.cc:824] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:31.531553 22735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312: No bootstrap required, opened a new log
I20260619 12:34:31.534013 22735 raft_consensus.cc:348] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } peers { permanent_uuid: "712eea69af18486396ca2ab357499ae7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } } peers { permanent_uuid: "56d55861702a49fdba9618ee9aa33ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } }
I20260619 12:34:31.534237 22735 raft_consensus.cc:729] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c34cf1fe5f1543d3b79fa044af87d312, State: Initialized, Role: FOLLOWER
I20260619 12:34:31.534642 22735 consensus_queue.cc:260] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } peers { permanent_uuid: "712eea69af18486396ca2ab357499ae7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } } peers { permanent_uuid: "56d55861702a49fdba9618ee9aa33ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } }
I20260619 12:34:31.535189 22921 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } peers { permanent_uuid: "712eea69af18486396ca2ab357499ae7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } } peers { permanent_uuid: "56d55861702a49fdba9618ee9aa33ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } } }
I20260619 12:34:31.535446 22921 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:34:31.536046 22735 sys_catalog.cc:564] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:31.536085 22709 raft_consensus.cc:482] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 12:34:31.536204 22709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:31.537581 22709 raft_consensus.cc:504] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } peers { permanent_uuid: "712eea69af18486396ca2ab357499ae7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } } peers { permanent_uuid: "56d55861702a49fdba9618ee9aa33ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } }
I20260619 12:34:31.538602 22709 leader_election.cc:288] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [CANDIDATE]: Term 1 election: Requested vote from peers 712eea69af18486396ca2ab357499ae7 (127.0.98.61:39621), 56d55861702a49fdba9618ee9aa33ecf (127.0.98.60:45343)
I20260619 12:34:31.539340 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34cf1fe5f1543d3b79fa044af87d312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "56d55861702a49fdba9618ee9aa33ecf"
I20260619 12:34:31.539597 22860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:31.540295 22784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c34cf1fe5f1543d3b79fa044af87d312" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "712eea69af18486396ca2ab357499ae7"
I20260619 12:34:31.540539 22784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:31.540905 22860 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c34cf1fe5f1543d3b79fa044af87d312 in term 1.
I20260619 12:34:31.541891 22685 leader_election.cc:302] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56d55861702a49fdba9618ee9aa33ecf, c34cf1fe5f1543d3b79fa044af87d312; no voters:
I20260619 12:34:31.541965 22784 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c34cf1fe5f1543d3b79fa044af87d312 in term 1.
I20260619 12:34:31.548615 22921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:31.548959 22921 raft_consensus.cc:686] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [term 1 LEADER]: Becoming Leader. State: Replica: c34cf1fe5f1543d3b79fa044af87d312, State: Running, Role: LEADER
I20260619 12:34:31.549413 22921 consensus_queue.cc:237] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } peers { permanent_uuid: "712eea69af18486396ca2ab357499ae7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } } peers { permanent_uuid: "56d55861702a49fdba9618ee9aa33ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } }
I20260619 12:34:31.550531 22667 master_runner.cc:263] No leader master found from master c34cf1fe5f1543d3b79fa044af87d312
I20260619 12:34:31.550602 22667 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 12:34:31.551540 22921 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c34cf1fe5f1543d3b79fa044af87d312. Latest consensus state: current_term: 1 leader_uuid: "c34cf1fe5f1543d3b79fa044af87d312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } peers { permanent_uuid: "712eea69af18486396ca2ab357499ae7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } } peers { permanent_uuid: "56d55861702a49fdba9618ee9aa33ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } } }
I20260619 12:34:31.551730 22921 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [sys.catalog]: This master's current role is: LEADER
W20260619 12:34:31.555785 22932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:34:31.557694 22932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:34:31.557824 22942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:31.558902 22942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 12:34:31.572110 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.544391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59224 (local address 127.0.98.60:45343)
0619 12:34:31.544498 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.544503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.545729 (+ 1226us) server_negotiation.cc:407] Connection header received
0619 12:34:31.545914 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.545918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.545971 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.546021 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.549803 (+ 3782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.550483 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.552821 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.553142 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.553229 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.556048 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:31.556053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:31.556055 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:31.556084 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:31.570119 (+ 14035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.570646 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.570650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.570652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.570743 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.571074 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.571078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.571080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.571304 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:31.571411 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:31.571794 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 12:34:31.571941 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 12:34:31.576354 22860 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf [term 1 FOLLOWER]: Refusing update from remote peer c34cf1fe5f1543d3b79fa044af87d312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:34:31.577134 22940 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56d55861702a49fdba9618ee9aa33ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:34:31.578312 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.545025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59392 (local address 127.0.98.61:39621)
0619 12:34:31.545783 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.545787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.552818 (+ 7031us) server_negotiation.cc:407] Connection header received
0619 12:34:31.553738 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.553752 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.553815 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.553877 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.565537 (+ 11660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.566317 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.567533 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.567906 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.567993 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.569870 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:31.569874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:31.569877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:31.569905 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:31.576189 (+ 6284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.576721 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.576725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.576727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.576827 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.577112 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.577116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.577117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.577320 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:31.577445 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:31.578049 (+ 604us) server_negotiation.cc:299] Negotiation successful
0619 12:34:31.578158 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683}
I20260619 12:34:31.578426 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:31.544036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58476 (local address 127.0.98.62:39849)
0619 12:34:31.544404 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.544410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.545813 (+ 1403us) server_negotiation.cc:407] Connection header received
0619 12:34:31.545859 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.545864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.545914 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.545983 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.547893 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.548605 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.551582 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:31.551940 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:31.552031 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:31.556801 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:31.556817 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:31.556820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:31.556859 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:31.574288 (+ 17429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.576905 (+ 2617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.576909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.576911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.577004 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:31.577477 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:31.577480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:31.577482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:31.577709 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:31.577829 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:31.578164 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 12:34:31.578285 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286}
I20260619 12:34:31.581574 22907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c34cf1fe5f1543d3b79fa044af87d312. Latest consensus state: current_term: 1 leader_uuid: "c34cf1fe5f1543d3b79fa044af87d312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } peers { permanent_uuid: "712eea69af18486396ca2ab357499ae7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } } peers { permanent_uuid: "56d55861702a49fdba9618ee9aa33ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } } }
I20260619 12:34:31.581703 22907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:34:31.582778 22784 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7 [term 1 FOLLOWER]: Refusing update from remote peer c34cf1fe5f1543d3b79fa044af87d312: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:34:31.583462 22940 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [LEADER]: Connected to new peer: Peer: permanent_uuid: "712eea69af18486396ca2ab357499ae7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:34:31.587358 22893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c34cf1fe5f1543d3b79fa044af87d312. Latest consensus state: current_term: 1 leader_uuid: "c34cf1fe5f1543d3b79fa044af87d312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } peers { permanent_uuid: "712eea69af18486396ca2ab357499ae7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } } peers { permanent_uuid: "56d55861702a49fdba9618ee9aa33ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } } }
I20260619 12:34:31.587491 22893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:34:31.589501 22907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c34cf1fe5f1543d3b79fa044af87d312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } peers { permanent_uuid: "712eea69af18486396ca2ab357499ae7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } } peers { permanent_uuid: "56d55861702a49fdba9618ee9aa33ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } } }
I20260619 12:34:31.589622 22907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:34:31.589803 22940 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c34cf1fe5f1543d3b79fa044af87d312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } peers { permanent_uuid: "712eea69af18486396ca2ab357499ae7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } } peers { permanent_uuid: "56d55861702a49fdba9618ee9aa33ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } } }
I20260619 12:34:31.589931 22940 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:31.590116 22944 mvcc.cc:205] Tried to move back new op lower bound from 7298549643567820800 to 7298549643470540800. Current Snapshot: MvccSnapshot[applied={T|T < 7298549643567820800}]
W20260619 12:34:31.592648 22839 tablet.cc:2369] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 12:34:31.594082 22763 tablet.cc:2369] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 12:34:31.594738 22942 catalog_manager.cc:1347] Generated new cluster ID: eeb7c81f422e43dfbcff1aea4201cd31
I20260619 12:34:31.594801 22942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:31.599054 22921 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c34cf1fe5f1543d3b79fa044af87d312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } peers { permanent_uuid: "712eea69af18486396ca2ab357499ae7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } } peers { permanent_uuid: "56d55861702a49fdba9618ee9aa33ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } } }
I20260619 12:34:31.599164 22921 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312 [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:31.600258 22893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c34cf1fe5f1543d3b79fa044af87d312" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c34cf1fe5f1543d3b79fa044af87d312" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } peers { permanent_uuid: "712eea69af18486396ca2ab357499ae7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39621 } } peers { permanent_uuid: "56d55861702a49fdba9618ee9aa33ecf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45343 } } }
I20260619 12:34:31.600373 22893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:34:31.601027 22947 mvcc.cc:205] Tried to move back new op lower bound from 7298549643567820800 to 7298549643470540800. Current Snapshot: MvccSnapshot[applied={T|T < 7298549643567820800}]
I20260619 12:34:31.603021 22945 mvcc.cc:205] Tried to move back new op lower bound from 7298549643567820800 to 7298549643470540800. Current Snapshot: MvccSnapshot[applied={T|T < 7298549643567820800}]
I20260619 12:34:31.619860 22942 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:31.621147 22942 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:31.637825 22942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c34cf1fe5f1543d3b79fa044af87d312: Generated new TSK 0
I20260619 12:34:31.638716 22942 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 12:34:31.666417 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:31.664975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59398 (local address 127.0.98.61:39621)
0619 12:34:31.665260 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.665267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.665482 (+ 215us) server_negotiation.cc:407] Connection header received
0619 12:34:31.665701 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.665704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.665772 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.665828 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.666287 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59398: BlockingRecv error: recv got EOF from 127.0.0.1:59398 (error 108)
Metrics: {"server-negotiator.queue_time_us":202}
W20260619 12:34:31.666417 22739 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:31.664838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58486 (local address 127.0.98.62:39849)
0619 12:34:31.665150 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.665157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.665349 (+ 192us) server_negotiation.cc:407] Connection header received
0619 12:34:31.665540 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.665544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.665592 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.665643 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.666291 (+ 648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58486: BlockingRecv error: recv got EOF from 127.0.0.1:58486 (error 108)
Metrics: {"server-negotiator.queue_time_us":190}
W20260619 12:34:31.667343 22891 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:31.665339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59234 (local address 127.0.98.60:45343)
0619 12:34:31.665468 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.665474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.666316 (+ 842us) server_negotiation.cc:407] Connection header received
0619 12:34:31.666474 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.666477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.666531 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.666586 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.667193 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59234: BlockingRecv error: recv got EOF from 127.0.0.1:59234 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 12:34:31.670358 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:31.669107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59404 (local address 127.0.98.61:39621)
0619 12:34:31.669209 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.669212 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.669346 (+ 134us) server_negotiation.cc:407] Connection header received
0619 12:34:31.669482 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.669484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.669541 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.669606 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.670097 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59404: BlockingRecv error: recv got EOF from 127.0.0.1:59404 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260619 12:34:31.673974 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:31.673213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.0.98.61:39621)
0619 12:34:31.673336 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.673341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.673354 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:34:31.673395 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.673400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.673446 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.673528 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.673884 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59416: BlockingRecv error: recv got EOF from 127.0.0.1:59416 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 12:34:31.678495 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:31.677560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59420 (local address 127.0.98.61:39621)
0619 12:34:31.677663 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.677667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.677792 (+ 125us) server_negotiation.cc:407] Connection header received
0619 12:34:31.677904 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.677906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.677941 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.678003 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.678391 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59420: BlockingRecv error: recv got EOF from 127.0.0.1:59420 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 12:34:31.685189 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:31.684111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59422 (local address 127.0.98.61:39621)
0619 12:34:31.684300 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.684304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.684341 (+ 37us) server_negotiation.cc:407] Connection header received
0619 12:34:31.684477 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.684479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.684516 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.684582 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.685040 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59422: BlockingRecv error: recv got EOF from 127.0.0.1:59422 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
W20260619 12:34:31.695904 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:31.694859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.0.98.61:39621)
0619 12:34:31.694984 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.694988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.695186 (+ 198us) server_negotiation.cc:407] Connection header received
0619 12:34:31.695283 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.695286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.695333 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.695419 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.695799 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59432: BlockingRecv error: recv got EOF from 127.0.0.1:59432 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 12:34:31.715319 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:31.714064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59448 (local address 127.0.98.61:39621)
0619 12:34:31.714196 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.714202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.714388 (+ 186us) server_negotiation.cc:407] Connection header received
0619 12:34:31.714523 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.714526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.714573 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.714659 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.715206 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59448: BlockingRecv error: recv got EOF from 127.0.0.1:59448 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 12:34:31.750895 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:31.749666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59458 (local address 127.0.98.61:39621)
0619 12:34:31.749794 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.749798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.749979 (+ 181us) server_negotiation.cc:407] Connection header received
0619 12:34:31.750106 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.750108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.750160 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.750234 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.750738 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59458: BlockingRecv error: recv got EOF from 127.0.0.1:59458 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 12:34:31.818392 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:31.817181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.0.98.61:39621)
0619 12:34:31.817343 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.817347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.817492 (+ 145us) server_negotiation.cc:407] Connection header received
0619 12:34:31.817609 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.817612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.817661 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.817746 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.818257 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59470: BlockingRecv error: recv got EOF from 127.0.0.1:59470 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 12:34:31.949846 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:31.948518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.0.98.61:39621)
0619 12:34:31.948672 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:31.948677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:31.948853 (+ 176us) server_negotiation.cc:407] Connection header received
0619 12:34:31.949035 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:31.949039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:31.949121 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:31.949198 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:31.949668 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59476: BlockingRecv error: recv got EOF from 127.0.0.1:59476 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 12:34:32.209296 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:32.208089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59484 (local address 127.0.98.61:39621)
0619 12:34:32.208254 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:32.208259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:32.208331 (+ 72us) server_negotiation.cc:407] Connection header received
0619 12:34:32.208463 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:32.208466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:32.208516 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:32.208601 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:32.209148 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59484: BlockingRecv error: recv got EOF from 127.0.0.1:59484 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 12:34:32.397500 22904 catalog_manager.cc:1259] Loaded cluster ID: eeb7c81f422e43dfbcff1aea4201cd31
I20260619 12:34:32.397590 22904 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7: loading cluster ID for follower catalog manager: success
I20260619 12:34:32.399307 22904 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7: acquiring CA information for follower catalog manager: success
I20260619 12:34:32.400517 22904 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 712eea69af18486396ca2ab357499ae7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:34:32.422556 22918 catalog_manager.cc:1259] Loaded cluster ID: eeb7c81f422e43dfbcff1aea4201cd31
I20260619 12:34:32.422649 22918 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf: loading cluster ID for follower catalog manager: success
I20260619 12:34:32.424427 22918 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf: acquiring CA information for follower catalog manager: success
I20260619 12:34:32.425649 22918 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 56d55861702a49fdba9618ee9aa33ecf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:34:32.727308 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:32.723409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.0.98.61:39621)
0619 12:34:32.723617 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:32.723621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:32.723731 (+ 110us) server_negotiation.cc:407] Connection header received
0619 12:34:32.723865 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:32.723867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:32.723993 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:32.724102 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:34:32.724365 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:32.724956 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:32.725869 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:32.726096 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:32.726186 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:32.726465 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:34:32.726957 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:34:32.727054 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:32.727130 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 12:34:32.727178 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":67,"threads_started":1}
I20260619 12:34:32.734371 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:32.730599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.0.98.60:45343)
0619 12:34:32.730833 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:32.730836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:32.730850 (+ 14us) server_negotiation.cc:407] Connection header received
0619 12:34:32.730980 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:32.730982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:32.731090 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:32.731186 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:34:32.731389 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:32.731920 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:32.732869 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:32.733161 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:32.733248 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:32.733514 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:34:32.733928 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:34:32.734021 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:32.734107 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 12:34:32.734156 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260619 12:34:32.742059 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:32.737554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59506 (local address 127.0.98.61:39621)
0619 12:34:32.737689 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:32.737695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:32.737928 (+ 233us) server_negotiation.cc:407] Connection header received
0619 12:34:32.738082 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:32.738087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:32.738143 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:32.738234 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:34:32.738620 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:32.739351 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:32.740442 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:32.740792 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:32.740898 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:32.741565 (+ 667us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:34:32.741654 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:34:32.741714 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:32.741876 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 12:34:32.741929 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:34:32.742158 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:32.738116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59262 (local address 127.0.98.60:45343)
0619 12:34:32.738259 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:32.738263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:32.738375 (+ 112us) server_negotiation.cc:407] Connection header received
0619 12:34:32.738554 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:32.738557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:32.738607 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:32.738691 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:34:32.738964 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:32.739760 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:32.740908 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:32.741214 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:32.741305 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:32.741652 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:34:32.741731 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:34:32.741774 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:32.741931 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 12:34:32.741988 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 12:34:32.742901 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:32.737550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58492 (local address 127.0.98.62:39849)
0619 12:34:32.737791 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:32.737796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:32.738334 (+ 538us) server_negotiation.cc:407] Connection header received
0619 12:34:32.738443 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:32.738446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:32.738618 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:32.738727 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:34:32.739081 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:32.739886 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:32.740951 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:32.741300 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:32.741404 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:32.741795 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:34:32.742160 (+ 365us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:34:32.742253 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:32.742656 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 12:34:32.742735 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":68,"threads_started":1}
I20260619 12:34:32.746079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 22667
I20260619 12:34:32.753346 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 22742
I20260619 12:34:32.759140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 22818
2026-06-19T12:34:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3237 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 12:34:32 dist-test-slave-fg3h krb5kdc[23034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 12:34:32 dist-test-slave-fg3h krb5kdc[23034](info): set up 2 sockets
Jun 19 12:34:32 dist-test-slave-fg3h krb5kdc[23034](info): commencing operation
krb5kdc: starting...
W20260619 12:34:33.805799 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/test-user.keytab.
Jun 19 12:34:33 dist-test-slave-fg3h krb5kdc[23034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T12:34:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T12:34:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 12:34:33.930171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35367
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35367,127.0.98.61:37059,127.0.98.60:33897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:34.041232 23047 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 12:34:34.041615 23047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:34.041692 23047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:34.041772 23047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:34.045856 23047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:34.045960 23047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:34.045998 23047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:34.046038 23047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:34.046073 23047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:34.051486 23047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35367,127.0.98.61:37059,127.0.98.60:33897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:34.052690 23047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:34.053817 23047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:34.063271 23052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:34.063288 23055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:34.063277 23053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:34.063637 23047 server_base.cc:1034] running on GCE node
I20260619 12:34:34.064440 23047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:34.065984 23047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:34.067167 23047 hybrid_clock.cc:648] HybridClock initialized: now 1781872474067085 us; error 102 us; skew 500 ppm
Jun 19 12:34:34 dist-test-slave-fg3h krb5kdc[23034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:34.070194 23047 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 12:34:34.071051 23047 webserver.cc:460] Webserver started at http://127.0.98.62:33085/ using document root <none> and password file <none>
I20260619 12:34:34.071575 23047 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:34.071633 23047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:34.071798 23047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:34.073872 23047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-0/data/instance:
uuid: "a7ff460410694effac399c8716a0529e"
format_stamp: "Formatted at 2026-06-19 12:34:34 on dist-test-slave-fg3h"
server_key: "b0d5477bbf61eac1b1c340fa718cc58a"
server_key_iv: "404d6a5ce7a64660a8e91f516ad420d5"
server_key_version: "encryptionkey@0"
I20260619 12:34:34.074396 23047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-0/wal/instance:
uuid: "a7ff460410694effac399c8716a0529e"
format_stamp: "Formatted at 2026-06-19 12:34:34 on dist-test-slave-fg3h"
server_key: "b0d5477bbf61eac1b1c340fa718cc58a"
server_key_iv: "404d6a5ce7a64660a8e91f516ad420d5"
server_key_version: "encryptionkey@0"
I20260619 12:34:34.078151 23047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 12:34:34.080595 23062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:34.081951 23047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 12:34:34.082079 23047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-0/wal
uuid: "a7ff460410694effac399c8716a0529e"
format_stamp: "Formatted at 2026-06-19 12:34:34 on dist-test-slave-fg3h"
server_key: "b0d5477bbf61eac1b1c340fa718cc58a"
server_key_iv: "404d6a5ce7a64660a8e91f516ad420d5"
server_key_version: "encryptionkey@0"
I20260619 12:34:34.082190 23047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:34.101455 23047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:34.105127 23047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:34.105368 23047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:34.112764 23047 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35367
I20260619 12:34:34.113147 23114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35367 every 8 connection(s)
I20260619 12:34:34.113871 23047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-0/data/info.pb
I20260619 12:34:34.116902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 23047
I20260619 12:34:34.116927 23115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:34.117038 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-0/wal/instance
I20260619 12:34:34.117383 392 external_mini_cluster.cc:1428] Setting key 9aff6d51954bc0eb9be96ad05ba6efa0
I20260619 12:34:34.121343 23115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 12:34:34 dist-test-slave-fg3h krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 12:34:34.139235 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.123853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35367 (local address 127.0.0.1:53816)
0619 12:34:34.124484 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.124520 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:34.125505 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:34.126400 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:34.126405 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:34.126868 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:34.127156 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:34.127170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.128797 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.128801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:34.130116 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:34.130131 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.131156 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.131159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:34.131264 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.131915 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:34.131932 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:34.133680 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:34.136130 (+ 2450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:34.136139 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:34.136147 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:34.136390 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:34.136794 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:34.136796 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:34.136797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:34.136911 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:34.137781 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:34.137819 (+ 38us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:34.137945 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 12:34:34.138091 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 12:34:34.138347 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"spinlock_wait_cycles":36608,"thread_start_us":176,"threads_started":1}
I20260619 12:34:34.139240 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.123853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53816 (local address 127.0.98.62:35367)
0619 12:34:34.124313 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.124335 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.124636 (+ 301us) server_negotiation.cc:407] Connection header received
0619 12:34:34.125888 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.125896 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.126164 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.126327 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.127333 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.128579 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.130267 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.130976 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.131177 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.133856 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:34.133866 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:34.133885 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:34.133919 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:34.136009 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.136530 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.136539 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.136543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.136690 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.137025 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.137030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.137033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.137615 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:34.137748 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:34.138414 (+ 666us) server_negotiation.cc:299] Negotiation successful
0619 12:34:34.138550 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":164,"threads_started":1}
I20260619 12:34:34.141649 23115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:34:34.142441 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.142193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37059 (local address 127.0.0.1:53802)
0619 12:34:34.142301 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.142374 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 12:34:34.142666 23063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37059: connect: Connection refused (error 111)
I20260619 12:34:34.143661 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.143474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37059 (local address 127.0.0.1:53808)
0619 12:34:34.143571 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.143613 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 12:34:34.143855 23115 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37059: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37059: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 12:34:34.146335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37059
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35367,127.0.98.61:37059,127.0.98.60:33897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 12:34:34.193177 23115 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } attempt: 1
I20260619 12:34:34.194115 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.193799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37059 (local address 127.0.0.1:53824)
0619 12:34:34.193967 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.194037 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 12:34:34.194878 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.194639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37059 (local address 127.0.0.1:53826)
0619 12:34:34.194744 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.194793 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 12:34:34.195006 23115 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37059: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37059: connect: Connection refused (error 111)
W20260619 12:34:34.256433 23122 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 12:34:34.256708 23122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:34.256731 23122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:34.256826 23122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:34.260830 23122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:34.260885 23122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:34.260905 23122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:34.260921 23122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:34.260983 23122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:34.263188 23115 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } attempt: 2
I20260619 12:34:34.263994 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.263694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37059 (local address 127.0.0.1:53842)
0619 12:34:34.263842 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.263912 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 12:34:34.264761 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.264555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37059 (local address 127.0.0.1:53846)
0619 12:34:34.264663 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.264709 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 12:34:34.264928 23115 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37059: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37059: connect: Connection refused (error 111)
I20260619 12:34:34.266656 23122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35367,127.0.98.61:37059,127.0.98.60:33897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:34.267711 23122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:34.268712 23122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:34.277292 23130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:34.277293 23128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:34.277348 23122 server_base.cc:1034] running on GCE node
W20260619 12:34:34.277293 23127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:34.278254 23122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:34.279394 23122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:34.280570 23122 hybrid_clock.cc:648] HybridClock initialized: now 1781872474280539 us; error 45 us; skew 500 ppm
Jun 19 12:34:34 dist-test-slave-fg3h krb5kdc[23034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:34.283548 23122 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 12:34:34.284480 23122 webserver.cc:460] Webserver started at http://127.0.98.61:39671/ using document root <none> and password file <none>
I20260619 12:34:34.285145 23122 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:34.285249 23122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:34.285491 23122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:34.287657 23122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-1/data/instance:
uuid: "20603d085a564d8ba7f5f98fb7e6c917"
format_stamp: "Formatted at 2026-06-19 12:34:34 on dist-test-slave-fg3h"
server_key: "edda803104e50fa58a0a75f41a06e62e"
server_key_iv: "fc82a0426dabd2c69ddd51dda9ee4ad9"
server_key_version: "encryptionkey@0"
I20260619 12:34:34.288237 23122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-1/wal/instance:
uuid: "20603d085a564d8ba7f5f98fb7e6c917"
format_stamp: "Formatted at 2026-06-19 12:34:34 on dist-test-slave-fg3h"
server_key: "edda803104e50fa58a0a75f41a06e62e"
server_key_iv: "fc82a0426dabd2c69ddd51dda9ee4ad9"
server_key_version: "encryptionkey@0"
I20260619 12:34:34.291988 23122 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 12:34:34.294564 23137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:34.295850 23122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 12:34:34.296003 23122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-1/wal
uuid: "20603d085a564d8ba7f5f98fb7e6c917"
format_stamp: "Formatted at 2026-06-19 12:34:34 on dist-test-slave-fg3h"
server_key: "edda803104e50fa58a0a75f41a06e62e"
server_key_iv: "fc82a0426dabd2c69ddd51dda9ee4ad9"
server_key_version: "encryptionkey@0"
I20260619 12:34:34.296124 23122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:34.320811 23122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:34.324270 23122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:34.324513 23122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:34.332019 23122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37059
I20260619 12:34:34.332309 23189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37059 every 8 connection(s)
I20260619 12:34:34.333199 23122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-1/data/info.pb
I20260619 12:34:34.335952 23190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:34.339605 23190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:34:34.342847 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 23122
I20260619 12:34:34.342989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-1/wal/instance
I20260619 12:34:34.343336 392 external_mini_cluster.cc:1428] Setting key c7f0aa1b2ecf258fa0205fde302ccc04
Jun 19 12:34:34 dist-test-slave-fg3h krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260619 12:34:34.355769 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.341474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53820 (local address 127.0.98.62:35367)
0619 12:34:34.341608 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.341612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.342420 (+ 808us) server_negotiation.cc:407] Connection header received
0619 12:34:34.343780 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.343783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.343849 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.343917 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.345511 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.346412 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.347973 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.348380 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.348482 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.351629 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:34.351632 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:34.351634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:34.351659 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:34.353363 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.353880 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.353883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.353885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.353992 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.354301 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.354303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.354304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.354584 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:34.354670 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:34.355349 (+ 679us) server_negotiation.cc:299] Negotiation successful
0619 12:34:34.355524 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 12:34:34.356098 23115 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } attempt: 3
I20260619 12:34:34.356680 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.341769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35367 (local address 127.0.0.1:53820)
0619 12:34:34.342264 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.342309 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:34.343509 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:34.344126 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:34.344138 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:34.344675 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:34.345351 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:34.345365 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.346559 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.346562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:34.347821 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:34.347831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.348567 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.348573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:34.348886 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.349575 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:34.349601 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:34.351437 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:34.353476 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:34.353482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:34.353493 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:34.353759 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:34.354088 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:34.354090 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:34.354092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:34.354221 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:34.354696 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:34.354705 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:34.355089 (+ 384us) client_negotiation.cc:769] Sending connection context
0619 12:34:34.355258 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 12:34:34.355522 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":117,"threads_started":1}
I20260619 12:34:34.358341 23190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 12:34:34 dist-test-slave-fg3h krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 12:34:34 dist-test-slave-fg3h krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 12:34:34.367290 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.356601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37059 (local address 127.0.0.1:53852)
0619 12:34:34.356784 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.356800 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:34.356896 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:34.357771 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:34.357774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:34.357949 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:34.358044 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:34.358049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.359277 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.359280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:34.359821 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:34.359829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.360441 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.360445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:34.360547 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.361042 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:34.361056 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:34.362456 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:34.365428 (+ 2972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:34.365432 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:34.365434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:34.365697 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:34.366021 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:34.366026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:34.366028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:34.366114 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:34.366746 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:34.366749 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:34.366850 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 12:34:34.366972 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 12:34:34.367114 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260619 12:34:34.367746 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.356715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53852 (local address 127.0.98.61:37059)
0619 12:34:34.357171 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.357184 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.357207 (+ 23us) server_negotiation.cc:407] Connection header received
0619 12:34:34.357241 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.357245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.357662 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.357797 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.358157 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.359163 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.359942 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.360323 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.360443 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.362587 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:34.362596 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:34.362610 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:34.362641 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:34.365316 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.365817 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.365823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.365825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.365926 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.366222 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.366225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.366227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.366579 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:34.366791 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:34.367077 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 12:34:34.367224 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":141,"threads_started":1}
I20260619 12:34:34.369902 23115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:34:34.370149 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.358848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37059 (local address 127.0.0.1:53854)
0619 12:34:34.359197 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.359214 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:34.359297 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:34.361545 (+ 2248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:34.361551 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:34.361740 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:34.361862 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:34.361870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.362976 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.362982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:34.363552 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:34.363561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.364065 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.364069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:34.364172 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.364560 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:34.364574 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:34.365903 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:34.368432 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:34.368436 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:34.368437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:34.368673 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:34.368968 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:34.368970 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:34.368972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:34.369050 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:34.369598 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:34.369604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:34.369712 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 12:34:34.369818 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 12:34:34.369942 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":13}
I20260619 12:34:34.370194 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.358959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53854 (local address 127.0.98.61:37059)
0619 12:34:34.361294 (+ 2335us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.361303 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.361318 (+ 15us) server_negotiation.cc:407] Connection header received
0619 12:34:34.361354 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.361358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.361398 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.361499 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.362002 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.362881 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.363662 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.363949 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.364030 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.366034 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:34.366044 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:34.366046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:34.366065 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:34.368307 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.368773 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.368776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.368778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.368869 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.369274 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.369277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.369279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.369483 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:34.369581 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:34.369891 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 12:34:34.370011 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2253,"thread_start_us":89,"threads_started":1}
I20260619 12:34:34.370647 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.370374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33897 (local address 127.0.0.1:42548)
0619 12:34:34.370504 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.370559 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 12:34:34.371483 23190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:34:34.371929 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.371367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33897 (local address 127.0.0.1:42552)
0619 12:34:34.371834 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.371877 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401}
I20260619 12:34:34.371966 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf39FOk/build/debug/bin/kudu
/tmp/dist-test-taskf39FOk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33897
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35367,127.0.98.61:37059,127.0.98.60:33897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 12:34:34.372071 23115 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33897: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33897: connect: Connection refused (error 111)
I20260619 12:34:34.372174 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.371906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33897 (local address 127.0.0.1:42554)
0619 12:34:34.372035 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.372111 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 12:34:34.372463 23140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33897: connect: Connection refused (error 111)
I20260619 12:34:34.373308 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.373130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33897 (local address 127.0.0.1:42568)
0619 12:34:34.373221 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.373261 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 12:34:34.373493 23190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33897: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33897: connect: Connection refused (error 111)
I20260619 12:34:34.403231 23115 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } attempt: 1
I20260619 12:34:34.404094 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.403795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33897 (local address 127.0.0.1:42570)
0619 12:34:34.403949 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.404011 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 12:34:34.404898 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.404649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33897 (local address 127.0.0.1:42576)
0619 12:34:34.404758 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.404843 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 12:34:34.405035 23115 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33897: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33897: connect: Connection refused (error 111)
I20260619 12:34:34.422812 23190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } attempt: 1
I20260619 12:34:34.423766 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.423447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33897 (local address 127.0.0.1:42584)
0619 12:34:34.423608 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.423684 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 12:34:34.424546 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.424321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33897 (local address 127.0.0.1:42596)
0619 12:34:34.424438 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.424491 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 12:34:34.424664 23190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33897: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33897: connect: Connection refused (error 111)
I20260619 12:34:34.480242 23115 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } attempt: 2
I20260619 12:34:34.481285 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.480836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33897 (local address 127.0.0.1:42600)
0619 12:34:34.481122 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.481199 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260619 12:34:34.482020 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.481803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33897 (local address 127.0.0.1:42616)
0619 12:34:34.481907 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.481959 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 12:34:34.482203 23115 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33897: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33897: connect: Connection refused (error 111)
W20260619 12:34:34.486904 23198 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 12:34:34.487234 23198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 12:34:34.487298 23198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 12:34:34.487375 23198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 12:34:34.491681 23198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 12:34:34.491771 23198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 12:34:34.491808 23198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 12:34:34.491852 23198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 12:34:34.491886 23198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 12:34:34.492830 23190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } attempt: 2
I20260619 12:34:34.493671 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.493375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33897 (local address 127.0.0.1:42622)
0619 12:34:34.493522 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.493594 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 12:34:34.494444 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.494225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33897 (local address 127.0.0.1:42626)
0619 12:34:34.494341 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.494393 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 12:34:34.494575 23190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33897: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33897: connect: Connection refused (error 111)
I20260619 12:34:34.498008 23198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35367,127.0.98.61:37059,127.0.98.60:33897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 11:53:36 UTC on ccycloud.quasar-bvtpgp.root.comops.site
build id 4455649
I20260619 12:34:34.499137 23198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 12:34:34.500231 23198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 12:34:34.509124 23204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 12:34:34.509150 23203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:34.509212 23198 server_base.cc:1034] running on GCE node
W20260619 12:34:34.509150 23206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 12:34:34.509987 23198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 12:34:34.511101 23198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 12:34:34.512267 23198 hybrid_clock.cc:648] HybridClock initialized: now 1781872474512254 us; error 30 us; skew 500 ppm
Jun 19 12:34:34 dist-test-slave-fg3h krb5kdc[23034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 12:34:34.515111 23198 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 12:34:34.516048 23198 webserver.cc:460] Webserver started at http://127.0.98.60:33871/ using document root <none> and password file <none>
I20260619 12:34:34.516635 23198 fs_manager.cc:359] Metadata directory not provided
I20260619 12:34:34.516790 23198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 12:34:34.517032 23198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 12:34:34.519227 23198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-2/data/instance:
uuid: "0ba18e073ee04b68a0e790fecda14867"
format_stamp: "Formatted at 2026-06-19 12:34:34 on dist-test-slave-fg3h"
server_key: "d687ff0fade7820d0c001d6cb378f151"
server_key_iv: "87bab553559c02118827a7d761479d71"
server_key_version: "encryptionkey@0"
I20260619 12:34:34.519807 23198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-2/wal/instance:
uuid: "0ba18e073ee04b68a0e790fecda14867"
format_stamp: "Formatted at 2026-06-19 12:34:34 on dist-test-slave-fg3h"
server_key: "d687ff0fade7820d0c001d6cb378f151"
server_key_iv: "87bab553559c02118827a7d761479d71"
server_key_version: "encryptionkey@0"
I20260619 12:34:34.523734 23198 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 12:34:34.526293 23213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 12:34:34.527737 23198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 12:34:34.527905 23198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-2/wal
uuid: "0ba18e073ee04b68a0e790fecda14867"
format_stamp: "Formatted at 2026-06-19 12:34:34 on dist-test-slave-fg3h"
server_key: "d687ff0fade7820d0c001d6cb378f151"
server_key_iv: "87bab553559c02118827a7d761479d71"
server_key_version: "encryptionkey@0"
I20260619 12:34:34.528076 23198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 12:34:34.547947 23198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 12:34:34.551369 23198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 12:34:34.551594 23198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 12:34:34.559262 23198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33897
I20260619 12:34:34.559551 23265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33897 every 8 connection(s)
I20260619 12:34:34.560272 23198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-2/data/info.pb
I20260619 12:34:34.563127 23266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 12:34:34.566900 23266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:34:34.568807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf39FOk/build/debug/bin/kudu as pid 23198
I20260619 12:34:34.568934 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf39FOk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781872188319340-392-0/minicluster-data/master-2/wal/instance
I20260619 12:34:34.569267 392 external_mini_cluster.cc:1428] Setting key fcadd52587cda827262a37469952db7b
Jun 19 12:34:34 dist-test-slave-fg3h krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 12:34:34 dist-test-slave-fg3h krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 12:34:34.580603 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.568757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.0.98.62:35367)
0619 12:34:34.568899 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.568904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.570006 (+ 1102us) server_negotiation.cc:407] Connection header received
0619 12:34:34.571047 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.571050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.571113 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.571202 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.572011 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.572513 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.573609 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.573837 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.573941 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.576628 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:34.576631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:34.576633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:34.576656 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:34.578242 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.578789 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.578792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.578794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.578876 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.579234 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.579238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.579240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.579500 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:34.579577 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:34.580291 (+ 714us) server_negotiation.cc:299] Negotiation successful
0619 12:34:34.580421 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:34:34.581362 23115 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } attempt: 3
I20260619 12:34:34.581773 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.569265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35367 (local address 127.0.0.1:53836)
0619 12:34:34.569845 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.569906 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:34.570835 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:34.571250 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:34.571255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:34.571606 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:34.571874 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:34.571886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.572641 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.572645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:34.573488 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:34.573497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.573977 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.573981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:34.574141 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.574731 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:34.574754 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:34.576455 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:34.578347 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:34.578354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:34.578364 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:34.578665 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:34.578969 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:34.578973 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:34.578975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:34.579144 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:34.579612 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:34.579620 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:34.579963 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 12:34:34.580202 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 12:34:34.580610 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"spinlock_wait_cycles":1920,"thread_start_us":148,"threads_started":1}
I20260619 12:34:34.581773 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.569940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42630 (local address 127.0.98.60:33897)
0619 12:34:34.570292 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.570303 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.570468 (+ 165us) server_negotiation.cc:407] Connection header received
0619 12:34:34.570884 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.570904 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.571101 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.571290 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.572083 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.572931 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.573804 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.574159 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.574280 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.577009 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:34.577018 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:34.577033 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:34.577061 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:34.578969 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.579459 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.579464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.579465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.579586 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.579900 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.579903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.579904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.580194 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:34.580384 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:34.580589 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 12:34:34.580719 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"spinlock_wait_cycles":1280,"thread_start_us":92,"threads_started":1}
I20260619 12:34:34.583384 23266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:34:34.585744 23190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } attempt: 3
Jun 19 12:34:34 dist-test-slave-fg3h krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 12:34:34 dist-test-slave-fg3h krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 12:34:34 dist-test-slave-fg3h krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 12:34:34 dist-test-slave-fg3h krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 12:34:34.592865 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.581828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42634 (local address 127.0.98.60:33897)
0619 12:34:34.582114 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.582123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.582135 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:34:34.582176 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.582181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.582224 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.582345 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.582897 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.583569 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.584498 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.584852 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.585338 (+ 486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.588210 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:34.588214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:34.588216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:34.588238 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:34.590364 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.591273 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.591278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.591281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.591390 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.591724 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.591727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.591729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.591947 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:34.592081 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:34.592506 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 12:34:34.592625 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":228,"thread_start_us":96,"threads_started":1}
I20260619 12:34:34.593997 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.581785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33897 (local address 127.0.0.1:42634)
0619 12:34:34.581946 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.581964 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:34.582038 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:34.582362 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:34.582365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:34.582531 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:34.582625 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:34.582631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.583706 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.583709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:34.584397 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:34.584408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.584976 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.584982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:34.585071 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.586539 (+ 1468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:34.586555 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:34.588074 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:34.590452 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:34.590456 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:34.590458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:34.590648 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:34.591507 (+ 859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:34.591511 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:34.591517 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:34.591613 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:34.592078 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:34.592083 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:34.592201 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 12:34:34.592331 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 12:34:34.593766 (+ 1435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"spinlock_wait_cycles":5376}
I20260619 12:34:34.595846 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.586203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42640 (local address 127.0.98.60:33897)
0619 12:34:34.586331 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.586335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.586411 (+ 76us) server_negotiation.cc:407] Connection header received
0619 12:34:34.586557 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.586560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.586597 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.587100 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.587170 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.588423 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.589461 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.589862 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.590326 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.592255 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:34.592261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:34.592266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:34.592293 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:34.594216 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.594738 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.594740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.594741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.594801 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.595070 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.595073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.595075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.595225 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:34.595322 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:34.595607 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 12:34:34.595710 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":57,"spinlock_wait_cycles":384}
I20260619 12:34:34.595847 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.586172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33897 (local address 127.0.0.1:42640)
0619 12:34:34.586357 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.586369 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:34.586452 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:34.586708 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:34.586713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:34.586906 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:34.586998 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:34.587004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.588516 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.588519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:34.589358 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:34.589372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.590006 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.590010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:34.590111 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.590804 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:34.590814 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:34.592121 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:34.594372 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:34.594377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:34.594380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:34.594627 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:34.594890 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:34.594893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:34.594895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:34.594978 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:34.595313 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:34.595317 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:34.595406 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 12:34:34.595549 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 12:34:34.595680 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260619 12:34:34.596720 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.584500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.0.98.62:35367)
0619 12:34:34.584598 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.584602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.584658 (+ 56us) server_negotiation.cc:407] Connection header received
0619 12:34:34.584766 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.584771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.584814 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.584893 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.585673 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.586393 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.587566 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.587930 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.588006 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.591601 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:34.591604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:34.591607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:34.591630 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:34.594524 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.595003 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.595006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.595007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.595088 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.595397 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.595401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.595402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.595592 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:34.595754 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:34.596490 (+ 736us) server_negotiation.cc:299] Negotiation successful
0619 12:34:34.596597 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 12:34:34.597661 23115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e: Bootstrap starting.
I20260619 12:34:34.598773 23190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917: Bootstrap starting.
I20260619 12:34:34.598922 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.583956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53856 (local address 127.0.98.61:37059)
0619 12:34:34.584073 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.584077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.584629 (+ 552us) server_negotiation.cc:407] Connection header received
0619 12:34:34.584787 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.584790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.584841 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.584896 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.585569 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.586401 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.587520 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.587914 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.587988 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.591580 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:34.591584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:34.591586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:34.591613 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:34.593409 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.597204 (+ 3795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.597208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.597209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.597290 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.597957 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.597960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.597962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.598158 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:34.598239 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:34.598668 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 12:34:34.598777 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 12:34:34.599534 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.583853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37059 (local address 127.0.0.1:53856)
0619 12:34:34.584566 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.584579 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:34.584687 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:34.585040 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:34.585043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:34.585245 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:34.585462 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:34.585471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.586494 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.586500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:34.587359 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:34.587367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.588126 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.588129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:34.588255 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.589183 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:34.589201 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:34.591417 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:34.593498 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:34.593501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:34.593503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:34.594024 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:34.597674 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:34.597677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:34.597678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:34.597843 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:34.598297 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:34.598299 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:34.598381 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 12:34:34.599264 (+ 883us) client_negotiation.cc:240] Negotiation successful
0619 12:34:34.599395 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":49}
I20260619 12:34:34.600344 23266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } has no permanent_uuid. Determining permanent_uuid...
I20260619 12:34:34.600615 23115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:34.601536 23115 log.cc:824] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:34.601737 23190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917: Neither blocks nor log segments found. Creating new log.
I20260619 12:34:34.602556 23190 log.cc:824] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:34.603654 23115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e: No bootstrap required, opened a new log
Jun 19 12:34:34 dist-test-slave-fg3h krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 12:34:34.604643 23190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917: No bootstrap required, opened a new log
Jun 19 12:34:34 dist-test-slave-fg3h krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781872474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 12:34:34.608211 23115 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ff460410694effac399c8716a0529e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } peers { permanent_uuid: "20603d085a564d8ba7f5f98fb7e6c917" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } } peers { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } }
I20260619 12:34:34.608544 23115 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ff460410694effac399c8716a0529e, State: Initialized, Role: FOLLOWER
I20260619 12:34:34.608902 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.598362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53868 (local address 127.0.98.61:37059)
0619 12:34:34.598467 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.598471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.598553 (+ 82us) server_negotiation.cc:407] Connection header received
0619 12:34:34.598716 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.598719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.598762 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.598821 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.599683 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.600400 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.601534 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.601890 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.602009 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.604800 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:34.604804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:34.604807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:34.604831 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:34.606561 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.607119 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.607123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.607125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.607204 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.607503 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.607506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.607508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.607728 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:34.607881 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:34.608612 (+ 731us) server_negotiation.cc:299] Negotiation successful
0619 12:34:34.608739 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 12:34:34.609172 23115 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ff460410694effac399c8716a0529e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } peers { permanent_uuid: "20603d085a564d8ba7f5f98fb7e6c917" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } } peers { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } }
I20260619 12:34:34.610734 23115 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:34.611112 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.600806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33897 (local address 127.0.0.1:42652)
0619 12:34:34.601152 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 12:34:34.601165 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 12:34:34.601233 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 12:34:34.603250 (+ 2017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 12:34:34.603252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 12:34:34.603426 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 12:34:34.603507 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:34.603514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.604412 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.604414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:34.605151 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 12:34:34.605162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.605767 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.605770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 12:34:34.605844 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.606234 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 12:34:34.606247 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 12:34:34.607614 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 12:34:34.609704 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:34.609707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:34.609708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:34.609917 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:34.610171 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 12:34:34.610173 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 12:34:34.610175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 12:34:34.610245 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 12:34:34.610635 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 12:34:34.610638 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 12:34:34.610708 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 12:34:34.610848 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 12:34:34.610969 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251}
I20260619 12:34:34.611028 23190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ff460410694effac399c8716a0529e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } peers { permanent_uuid: "20603d085a564d8ba7f5f98fb7e6c917" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } } peers { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } }
I20260619 12:34:34.611249 23190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20603d085a564d8ba7f5f98fb7e6c917, State: Initialized, Role: FOLLOWER
I20260619 12:34:34.611567 23190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ff460410694effac399c8716a0529e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } peers { permanent_uuid: "20603d085a564d8ba7f5f98fb7e6c917" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } } peers { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } }
I20260619 12:34:34.611603 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.601026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.0.98.60:33897)
0619 12:34:34.603004 (+ 1978us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.603009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.603021 (+ 12us) server_negotiation.cc:407] Connection header received
0619 12:34:34.603054 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.603056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.603094 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.603167 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.603620 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.604292 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.605267 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.605589 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.605675 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.607757 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:34.607761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:34.607763 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:34.607784 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:34.609601 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.610013 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.610015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.610017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.610083 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.610344 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.610346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.610348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.610518 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:34.610604 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:34.611360 (+ 756us) server_negotiation.cc:299] Negotiation successful
0619 12:34:34.611460 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1914}
I20260619 12:34:34.612202 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 12:34:34.612818 23190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:34.614797 23266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867: Bootstrap starting.
W20260619 12:34:34.617528 23288 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 12:34:34.617542 23266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867: Neither blocks nor log segments found. Creating new log.
W20260619 12:34:34.617594 23288 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:34:34.618389 23266 log.cc:824] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867: Log is configured to *not* fsync() on all Append() calls
I20260619 12:34:34.619068 23089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a7ff460410694effac399c8716a0529e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53850
I20260619 12:34:34.619261 23089 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 12:34:34.619370 23089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:34.620630 23089 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ff460410694effac399c8716a0529e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } peers { permanent_uuid: "20603d085a564d8ba7f5f98fb7e6c917" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } } peers { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } }
I20260619 12:34:34.621526 23089 leader_election.cc:288] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [CANDIDATE]: Term 1 election: Requested vote from peers 20603d085a564d8ba7f5f98fb7e6c917 (127.0.98.61:37059), 0ba18e073ee04b68a0e790fecda14867 (127.0.98.60:33897)
I20260619 12:34:34.622306 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ff460410694effac399c8716a0529e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } peers { permanent_uuid: "20603d085a564d8ba7f5f98fb7e6c917" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } } peers { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } } }
I20260619 12:34:34.622462 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:34:34.619213 23122 master_runner.cc:186] Error getting master registration for 127.0.98.60:33897: OK, instance_id { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" instance_seqno: 1781872474557050 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 12:34:34.618677 23047 master_runner.cc:186] Error getting master registration for 127.0.98.60:33897: OK, instance_id { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" instance_seqno: 1781872474557050 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 12:34:34.622597 23122 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 12:34:34.622614 23047 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 12:34:34.622786 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ff460410694effac399c8716a0529e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } peers { permanent_uuid: "20603d085a564d8ba7f5f98fb7e6c917" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } } peers { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } } }
I20260619 12:34:34.622951 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:34:34.623904 23164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7ff460410694effac399c8716a0529e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "20603d085a564d8ba7f5f98fb7e6c917"
I20260619 12:34:34.624120 23164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:34.625509 23164 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7ff460410694effac399c8716a0529e in term 1.
I20260619 12:34:34.625902 23063 leader_election.cc:302] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20603d085a564d8ba7f5f98fb7e6c917, a7ff460410694effac399c8716a0529e; no voters:
I20260619 12:34:34.626065 23266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867: No bootstrap required, opened a new log
W20260619 12:34:34.626123 23297 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:34:34.626179 23297 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:34:34.626211 23274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 12:34:34.626467 23274 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [term 1 LEADER]: Becoming Leader. State: Replica: a7ff460410694effac399c8716a0529e, State: Running, Role: LEADER
I20260619 12:34:34.626821 23274 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ff460410694effac399c8716a0529e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } peers { permanent_uuid: "20603d085a564d8ba7f5f98fb7e6c917" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } } peers { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } }
I20260619 12:34:34.628975 23307 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7ff460410694effac399c8716a0529e. Latest consensus state: current_term: 1 leader_uuid: "a7ff460410694effac399c8716a0529e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ff460410694effac399c8716a0529e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } peers { permanent_uuid: "20603d085a564d8ba7f5f98fb7e6c917" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } } peers { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } } }
I20260619 12:34:34.629141 23307 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:34.630317 23266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ff460410694effac399c8716a0529e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } peers { permanent_uuid: "20603d085a564d8ba7f5f98fb7e6c917" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } } peers { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } }
I20260619 12:34:34.630571 23266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba18e073ee04b68a0e790fecda14867, State: Initialized, Role: FOLLOWER
I20260619 12:34:34.630903 23266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ff460410694effac399c8716a0529e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } peers { permanent_uuid: "20603d085a564d8ba7f5f98fb7e6c917" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } } peers { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } }
I20260619 12:34:34.631582 23310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ff460410694effac399c8716a0529e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } peers { permanent_uuid: "20603d085a564d8ba7f5f98fb7e6c917" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } } peers { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } } }
I20260619 12:34:34.631713 23310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:34:34.632553 23266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 12:34:34.632571 23240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7ff460410694effac399c8716a0529e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0ba18e073ee04b68a0e790fecda14867"
I20260619 12:34:34.632853 23240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867 [term 0 FOLLOWER]: Advancing to term 1
I20260619 12:34:34.633621 23309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 12:34:34.634204 23240 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7ff460410694effac399c8716a0529e in term 1.
I20260619 12:34:34.634672 23309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 12:34:34.637836 23322 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 12:34:34.637954 23322 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 12:34:34.638402 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.623980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.0.98.61:37059)
0619 12:34:34.624087 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.624093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.624970 (+ 877us) server_negotiation.cc:407] Connection header received
0619 12:34:34.625275 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.625278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.625340 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.625401 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.627107 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.627882 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.629063 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.629438 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.629516 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.631450 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:34.631454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:34.631457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:34.631482 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:34.633555 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.634859 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.634863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.634865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.634949 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.635502 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.635506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.635508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.635726 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:34.635825 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:34.636977 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0619 12:34:34.638227 (+ 1250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 12:34:34.653131 23240 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867 [term 1 FOLLOWER]: Refusing update from remote peer a7ff460410694effac399c8716a0529e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:34:34.653262 23164 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917 [term 1 FOLLOWER]: Refusing update from remote peer a7ff460410694effac399c8716a0529e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 12:34:34.653820 23274 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:34:34.654279 23274 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20603d085a564d8ba7f5f98fb7e6c917" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 12:34:34.654424 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.623896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.0.98.60:33897)
0619 12:34:34.624010 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.624015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.639962 (+ 15947us) server_negotiation.cc:407] Connection header received
0619 12:34:34.640127 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.640130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.640176 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.640226 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.641657 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.642410 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.643528 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.643850 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.643923 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.645744 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:34.645748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:34.645752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:34.645776 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:34.647549 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.648075 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.648079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.648081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.648162 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.648461 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.648465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.648467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.648675 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:34.648812 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:34.654149 (+ 5337us) server_negotiation.cc:299] Negotiation successful
0619 12:34:34.654271 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 12:34:34.658321 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7ff460410694effac399c8716a0529e. Latest consensus state: current_term: 1 leader_uuid: "a7ff460410694effac399c8716a0529e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ff460410694effac399c8716a0529e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } peers { permanent_uuid: "20603d085a564d8ba7f5f98fb7e6c917" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } } peers { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } } }
I20260619 12:34:34.658447 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:34:34.661327 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:34.632964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53856 (local address 127.0.98.62:35367)
0619 12:34:34.633109 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.633114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.644550 (+ 11436us) server_negotiation.cc:407] Connection header received
0619 12:34:34.645295 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.645299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.645356 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.645405 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.646678 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.647507 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.655454 (+ 7947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:34.655758 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:34.655835 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:34.657696 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 12:34:34.657700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 12:34:34.657703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 12:34:34.657729 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 12:34:34.659529 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.660026 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.660030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.660032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.660110 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 12:34:34.660390 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 12:34:34.660394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 12:34:34.660396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 12:34:34.660602 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 12:34:34.660709 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:34.661035 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 12:34:34.661173 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 12:34:34.662223 23310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7ff460410694effac399c8716a0529e. Latest consensus state: current_term: 1 leader_uuid: "a7ff460410694effac399c8716a0529e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ff460410694effac399c8716a0529e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } peers { permanent_uuid: "20603d085a564d8ba7f5f98fb7e6c917" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } } peers { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } } }
I20260619 12:34:34.662352 23310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:34:34.674846 23307 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7ff460410694effac399c8716a0529e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ff460410694effac399c8716a0529e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } peers { permanent_uuid: "20603d085a564d8ba7f5f98fb7e6c917" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } } peers { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } } }
I20260619 12:34:34.674988 23307 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:34.675099 23324 mvcc.cc:205] Tried to move back new op lower bound from 7298549656170500096 to 7298549656074952704. Current Snapshot: MvccSnapshot[applied={T|T < 7298549656170500096}]
I20260619 12:34:34.675969 23310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7ff460410694effac399c8716a0529e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ff460410694effac399c8716a0529e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } peers { permanent_uuid: "20603d085a564d8ba7f5f98fb7e6c917" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } } peers { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } } }
I20260619 12:34:34.676088 23310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:34:34.676805 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7ff460410694effac399c8716a0529e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ff460410694effac399c8716a0529e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } peers { permanent_uuid: "20603d085a564d8ba7f5f98fb7e6c917" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } } peers { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } } }
I20260619 12:34:34.676921 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e [sys.catalog]: This master's current role is: LEADER
I20260619 12:34:34.677913 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7ff460410694effac399c8716a0529e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ff460410694effac399c8716a0529e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35367 } } peers { permanent_uuid: "20603d085a564d8ba7f5f98fb7e6c917" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37059 } } peers { permanent_uuid: "0ba18e073ee04b68a0e790fecda14867" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33897 } } }
I20260619 12:34:34.678030 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 12:34:34.679621 23327 mvcc.cc:205] Tried to move back new op lower bound from 7298549656170500096 to 7298549656074952704. Current Snapshot: MvccSnapshot[applied={T|T < 7298549656170500096}]
I20260619 12:34:34.690748 23309 catalog_manager.cc:1347] Generated new cluster ID: 9476b58ad28b4c7390360c321f0ce283
I20260619 12:34:34.690837 23309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 12:34:34.720175 23309 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 12:34:34.721517 23309 catalog_manager.cc:1504] Loading token signing keys...
I20260619 12:34:34.743928 23309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7ff460410694effac399c8716a0529e: Generated new TSK 0
I20260619 12:34:34.744879 23309 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 12:34:34.775938 23269 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:34.773285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42670 (local address 127.0.98.60:33897)
0619 12:34:34.773431 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.773436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.774916 (+ 1480us) server_negotiation.cc:407] Connection header received
0619 12:34:34.775104 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.775107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.775156 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.775206 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.775797 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42670: BlockingRecv error: recv got EOF from 127.0.0.1:42670 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 12:34:34.777318 23118 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:34.773765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.0.98.62:35367)
0619 12:34:34.773887 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.773892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.776397 (+ 2505us) server_negotiation.cc:407] Connection header received
0619 12:34:34.776562 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.776565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.776615 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.776666 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.777203 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53870: BlockingRecv error: recv got EOF from 127.0.0.1:53870 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 12:34:34.778736 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:34.774415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.0.98.61:37059)
0619 12:34:34.774519 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.774524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.777755 (+ 3231us) server_negotiation.cc:407] Connection header received
0619 12:34:34.778034 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.778038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.778086 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.778136 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.778595 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53884: BlockingRecv error: recv got EOF from 127.0.0.1:53884 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260619 12:34:34.783046 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:34.781584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.0.98.61:37059)
0619 12:34:34.781690 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.781694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.782121 (+ 427us) server_negotiation.cc:407] Connection header received
0619 12:34:34.782325 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.782328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.782399 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.782451 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.782947 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53896: BlockingRecv error: recv got EOF from 127.0.0.1:53896 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 12:34:34.788285 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:34.786770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.0.98.61:37059)
0619 12:34:34.786888 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.786893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.787359 (+ 466us) server_negotiation.cc:407] Connection header received
0619 12:34:34.787548 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.787551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.787604 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.787676 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.788189 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53906: BlockingRecv error: recv got EOF from 127.0.0.1:53906 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 12:34:34.798650 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:34.793496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.0.98.61:37059)
0619 12:34:34.793626 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.793631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.797542 (+ 3911us) server_negotiation.cc:407] Connection header received
0619 12:34:34.797785 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.797789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.797859 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.797918 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.798527 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53908: BlockingRecv error: recv got EOF from 127.0.0.1:53908 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 12:34:34.807507 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:34.805554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53924 (local address 127.0.98.61:37059)
0619 12:34:34.805710 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.805715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.806387 (+ 672us) server_negotiation.cc:407] Connection header received
0619 12:34:34.806615 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.806619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.806676 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.806753 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.807385 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53924: BlockingRecv error: recv got EOF from 127.0.0.1:53924 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 12:34:34.822618 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:34.818227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53926 (local address 127.0.98.61:37059)
0619 12:34:34.818354 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.818359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.821458 (+ 3099us) server_negotiation.cc:407] Connection header received
0619 12:34:34.821726 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.821731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.821795 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.821860 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.822492 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53926: BlockingRecv error: recv got EOF from 127.0.0.1:53926 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 12:34:34.843657 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:34.841590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53942 (local address 127.0.98.61:37059)
0619 12:34:34.841750 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.841756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.842463 (+ 707us) server_negotiation.cc:407] Connection header received
0619 12:34:34.842703 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.842709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.842774 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.842832 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.843516 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53942: BlockingRecv error: recv got EOF from 127.0.0.1:53942 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 12:34:34.879829 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:34.878450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53956 (local address 127.0.98.61:37059)
0619 12:34:34.878777 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.878780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.878838 (+ 58us) server_negotiation.cc:407] Connection header received
0619 12:34:34.878971 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.878973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.879020 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.879100 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.879672 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53956: BlockingRecv error: recv got EOF from 127.0.0.1:53956 (error 108)
Metrics: {"server-negotiator.queue_time_us":236}
W20260619 12:34:34.947188 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:34.945898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53968 (local address 127.0.98.61:37059)
0619 12:34:34.946057 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:34.946064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:34.946255 (+ 191us) server_negotiation.cc:407] Connection header received
0619 12:34:34.946432 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:34.946435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:34.946483 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:34.946572 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:34.947080 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53968: BlockingRecv error: recv got EOF from 127.0.0.1:53968 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 12:34:35.078754 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:35.077616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53976 (local address 127.0.98.61:37059)
0619 12:34:35.077747 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:35.077751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:35.077955 (+ 204us) server_negotiation.cc:407] Connection header received
0619 12:34:35.078061 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:35.078064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:35.078109 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:35.078192 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:35.078628 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53976: BlockingRecv error: recv got EOF from 127.0.0.1:53976 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 12:34:35.337981 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 12:34:35.336735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53980 (local address 127.0.98.61:37059)
0619 12:34:35.336874 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:35.336878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:35.337122 (+ 244us) server_negotiation.cc:407] Connection header received
0619 12:34:35.337276 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:35.337279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:35.337336 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:35.337427 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 12:34:35.337876 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53980: BlockingRecv error: recv got EOF from 127.0.0.1:53980 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 12:34:35.627487 23297 catalog_manager.cc:1259] Loaded cluster ID: 9476b58ad28b4c7390360c321f0ce283
I20260619 12:34:35.627638 23297 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917: loading cluster ID for follower catalog manager: success
I20260619 12:34:35.629349 23297 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917: acquiring CA information for follower catalog manager: success
I20260619 12:34:35.630491 23297 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 20603d085a564d8ba7f5f98fb7e6c917: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:34:35.639120 23322 catalog_manager.cc:1259] Loaded cluster ID: 9476b58ad28b4c7390360c321f0ce283
I20260619 12:34:35.639216 23322 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867: loading cluster ID for follower catalog manager: success
I20260619 12:34:35.640815 23322 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867: acquiring CA information for follower catalog manager: success
I20260619 12:34:35.641999 23322 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0ba18e073ee04b68a0e790fecda14867: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 12:34:35.856426 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:35.852280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.0.98.61:37059)
0619 12:34:35.852554 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:35.852558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:35.852575 (+ 17us) server_negotiation.cc:407] Connection header received
0619 12:34:35.852631 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:35.852633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:35.852797 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:35.852918 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:34:35.853187 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:35.853789 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:35.854873 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:35.855109 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:35.855203 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:35.855527 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:34:35.856021 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:34:35.856117 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:35.856237 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 12:34:35.856296 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":80,"threads_started":1}
I20260619 12:34:35.863731 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:35.859836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.0.98.60:33897)
0619 12:34:35.860100 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:35.860104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:35.860117 (+ 13us) server_negotiation.cc:407] Connection header received
0619 12:34:35.860246 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:35.860248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:35.860371 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:35.860470 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:34:35.860664 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:35.861298 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:35.862142 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:35.862368 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:35.862453 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:35.862797 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:34:35.863275 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:34:35.863425 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:35.863470 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 12:34:35.863547 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":92,"threads_started":1}
I20260619 12:34:35.870409 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:35.866801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53994 (local address 127.0.98.61:37059)
0619 12:34:35.866950 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:35.866955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:35.867099 (+ 144us) server_negotiation.cc:407] Connection header received
0619 12:34:35.867253 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:35.867256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:35.867308 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:35.867388 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:34:35.867630 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:35.868216 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:35.869189 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:35.869537 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:35.869635 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:35.869921 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:34:35.869999 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:34:35.870042 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:35.870214 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 12:34:35.870272 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 12:34:35.871031 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:35.866628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42682 (local address 127.0.98.60:33897)
0619 12:34:35.866729 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:35.866734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:35.867332 (+ 598us) server_negotiation.cc:407] Connection header received
0619 12:34:35.867378 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:35.867380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:35.867423 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:35.867490 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:34:35.867931 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:35.868682 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:35.869888 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:35.870116 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:35.870238 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:35.870629 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:34:35.870690 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:34:35.870777 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:35.870861 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 12:34:35.870908 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 12:34:35.871547 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 12:34:35.866577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.0.98.62:35367)
0619 12:34:35.866882 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 12:34:35.866909 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 12:34:35.867081 (+ 172us) server_negotiation.cc:407] Connection header received
0619 12:34:35.867264 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 12:34:35.867271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 12:34:35.867433 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 12:34:35.867549 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 12:34:35.867858 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:35.868690 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:35.869730 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 12:34:35.870071 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 12:34:35.870186 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 12:34:35.870498 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 12:34:35.871156 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 12:34:35.871240 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 12:34:35.871333 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 12:34:35.871381 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":74,"threads_started":1}
I20260619 12:34:35.874509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 23047
I20260619 12:34:35.881362 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 23122
I20260619 12:34:35.887483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf39FOk/build/debug/bin/kudu with pid 23198
2026-06-19T12:34:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3127 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6365 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (287560 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST